Method of describing object region data, apparatus for generating object region data, video processing apparatus and video processing method

ABSTRACT

A region data describing method for describing, over a plurality of frames, region data about the region of an arbitrary object in a video, the method specifying the object region in the video with at least either of an approximate figure approximating the region or characteristic points of the region, approximating a trajectory obtained by arranging position data of the representative points or the characteristic point in a direction in which frames proceed with a predetermined function and describing the parameter of the function as region data. Thus, the region of a predetermined object in the video can be described with a small quantity of data. Moreover, creation and handling of data can easily be performed.

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is based upon and claims the benefit of priorityfrom the prior Japanese Patent Application No. 11-187033, filed Jun. 30,1999, the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

[0002] The present invention relates to a method of describing objectregion data such that information about an object region in a video isdescribed, an apparatus for generating object region data such thatinformation about an object region in a video is generated, a videoprocessing apparatus arranged to be given an instruction about an objectin a video to perform a predetermined process or retrieve an object in avideo, and a video processing method therefor.

[0003] Hyper media are configured such that related information called ahyper link is given in between mediums, such as videos, sounds or texts,to permit mutual reference. When videos are mainly used, relatedinformation has been provided for each object which appears in thevideo. When the object is specified, related information (textinformation or the like) is displayed. The foregoing structure is arepresentative example of the hyper media. The object in the video isexpressed by a frame number or a time stamp of the video, andinformation for identifying a region in the video which are recorded invideo data or recorded as individual data.

[0004] Mask images have frequently been used as means for identifying aregion in a video. The mask image is a bit map image constituted bygiving different pixel values between the inside portion of anidentified region and the outside portion of the same. A simplest methodhas an arrangement that a pixel value of “1” is given to the insideportion of the region and “0” is given to the outside portion of thesame. Alternatively, a values which are employed in computer graphicsare sometimes employed. Since the a value is usually able to express 256levels of gray, a portion of the levels is used. The inside portion ofthe specified region is expressed as 255, while the outside portion ofthe same is expressed as 0. The latter image is called an a map. Whenthe regions in the image are expressed by the mask images, determinationwhether or not a pixel in a frame is included in the specified regioncan easily be made by reading the value of the pixel of the mask imageand by determining whether the value is 0 or 255. The mask image hasfreedom with which a region can be expressed regardless of the shape ofthe region and even a discontinuous region can be expressed. The maskimage must have pixels, the size of which is the same as the size of theoriginal image. Thus, there arises a problem in that the quantity ofdata cannot be reduced.

[0005] To reduce the quantity of data of the mask image, the mask imageis frequently compressed. When the mask image is a binary mask imageconstituted by 0 and 1, a process of a binary image can be performed.Therefore, the compression method employed in facsimile machines or thelike is frequently employed. In the case of MPEG-4 in which ISO/IEC MPEG(Moving Picture Experts Group) has been standardized, an arbitrary shapecoding method will be employed in which the mask image constituted by 0and 1 and the mask image using the a value are compressed. The foregoingcompression method is a method using motion compensation and capable ofimproving compression efficiency. On the other hand, complex compressionand decoding processes are required.

[0006] To express a region in a video, the mask image or the compressedmask image has usually been employed. However, data for identifying aregion is required to permit easy and quick extraction, to be reduced inquantity and to permit easy handling.

[0007] On the other hand, the hyper media, which are usually assumedthat an operation for displaying related information of a moving objectin a video is performed, have somewhat difficulty in specifying theobject as distinct from handling of a still image. A user usually hasdifficulty in specifying a specific portion. Therefore, it can beconsidered that the user usually aims, for example, a portion in thevicinity of the center of the object in a rough manner. Moreover, aportion adjacent to the object which is deviated from the object isfrequently specified according to the movement of the object. Therefore,data for specifying a region is desired to be adaptable to the foregoingmedia. Moreover, an aiding mechanism for facilitating specification of amoving object in a video is required for the system for displayingrelated information of the moving object in the video.

[0008] As described above, the conventional method of expressing adesired object region in a video by using the mask image suffers from aproblem in that the quantity of data cannot be reduced. The methodarranged to compress the mask image raises a problem in that coding anddecoding become too complicated. What is worse, directly accessing tothe pixel of a predetermined frame cannot be performed, causing handlingto become difficult.

[0009] There arises another problem in that a device for permitting auser to easily instruct a moving object in a video has not beenprovided.

BRIEF SUMMARY OF THE INVENTION

[0010] Accordingly, it is an object of the present invention to providea method of describing object region data and an apparatus forgenerating object region data which are capable of describing a desiredobject region in a video by using a small quantity of data andfacilitating generation of data and handling of the same.

[0011] Another object of the present invention is to provide a method ofdescribing object region data, an apparatus for generating object regiondata, a video processing method and a video processing apparatus withwhich a user is permitted to easily instruct an object in a video anddetermine the object.

[0012] Another object of the present invention is to provide a method ofdescribing object region data, an apparatus for generating object regiondata, a video processing method and a video processing apparatus withwhich retrieval of an object in a video can easily be performed.

[0013] According to one aspect of the present invention, there isprovided a method of describing object region data such that informationabout an arbitrary object region in a video is described over aplurality of continuous frames, the method identifying a desired objectregion in a video according to at least either of a figure approximatedto the object region or a characteristic point of the object region;approximating a trajectory obtained by arranging positions ofrepresentative points of the approximate figure or the characteristicpoints of the object region in a direction in which frames proceed witha predetermined function; and describing information about the objectregion by using the parameter of the function.

[0014] According to another aspect of the present invention, there isprovided a method of describing object region data such that informationabout an arbitrary object region in a video is described over aplurality of continuous frames, the method describing the object regiondata by using information capable of identifying at least the framenumber of a leading frame and the frame number of a trailing frame ofthe plurality of the subject frames or the time stamp of the leadingframe and the time stamp of the trailing frame, information foridentifying the type of the figure of an approximate figureapproximating the object region, and the parameter of a function withwhich a trajectory obtained by arranging position data of representativepoints of the approximate figure corresponding to the object region in adirection in which frames proceed has been approximated.

[0015] According to another aspect of the present invention, there isprovided a method of describing object region data such that informationabout an arbitrary object region in a video is described over aplurality of continuous frames, the method describing the object regiondata by using information capable of identifying at least the framenumber of a leading frame and the frame number of a trailing frame ofthe plurality of the subject frames or the time stamp of the leadingframe and the time stamp of the trailing frame, the number ofapproximate figures approximating the object region, information foridentifying the type of the figure of an approximate figure and theparameters of functions with which trajectories corresponding to theapproximate figures and obtained by arranging position data ofrepresentative points of each approximate figure in a direction in whichframes proceed have been approximated.

[0016] According to another aspect of the present invention, there isprovided a method of describing object region data such that informationabout an arbitrary object region in a video is described over aplurality of continuous frames, the method describing the object regiondata by using information capable of identifying at least the framenumber of a leading frame and the frame number of a trailing frame ofthe plurality of the subject frames or the time stamp of the leadingframe and the time stamp of the trailing frame, and the parameter of afunction with which a trajectory obtained by arranging position data ofcharacteristic points of the object region in a direction in whichframes proceed has been approximated.

[0017] Information capable of identifying the frame number of a leadingframe and the frame number of a trailing frame of the plurality of thesubject frames or the time stamp of the leading frame and the time stampof the trailing frame is the leading frame number and a trailing framenumber or the leading frame number and the difference between theleading frame number and the trailing frame number.

[0018] The parameter of the function may be position data of knots ofthe trajectory and information arranged to be used together with theposition data of the knots to be capable of identifying the trajectory.Alternatively, the parameter of the function may be a coefficient of thefunction.

[0019] When a plurality of representative points of the approximatefigure of the object region or characteristic points of the objectregion exist, it is desirable to identify the correspondence between theplural representative points or the characteristic points of the presentframe and a plurality of representative points or characteristic pointsof an adjacent frame.

[0020] It is desirable to describe information related to the object ora method of accessing to the related information.

[0021] According to another aspect of the present invention, there isprovided a recording medium storing object region data containinginformation about regions of one or more objects described by one of theabove methods.

[0022] According to another aspect of the present invention, there isprovided a recording medium storing object region data containinginformation about regions of one or more objects described by one of theabove methods and information related to each object or informationindicating a method of accessing to the related information.

[0023] According to another aspect of the present invention, there isprovided a recording medium storing object region data containinginformation about regions of one or more objects described by one of theabove methods and information for identifying information related toeach object, and information related to each object.

[0024] According to another aspect of the present invention, there isprovided a video processing method for determining whether or not apredetermined object has been specified in a screen which is displayinga video, the method obtaining information describing parameter of afunction approximating a trajectory obtained by arranging position dataof representative points of the approximate figure in a direction inwhich frames proceed when an arbitrary position has been specified inthe screen in a case where a region of the predetermined object existsin the video; detecting the position of the representative point in theframe based on the obtained information; detecting the position of theapproximate figure in accordance with the detected position of therepresentative point; determining whether or not the input positionexists in the approximate figure; and determining that the predeterminedobject has been specified when a determination has been made that theinput position exists in the approximate figure.

[0025] According to another aspect of the present invention, there isprovided a video processing method for determining whether or not apredetermined object has been specified in a screen which is displayinga video, the method obtaining information describing parameter of afunction approximating a trajectory obtained by arranging position dataof characteristic points of the object region in a direction in whichframes proceed when an arbitrary position has been specified in thescreen in a case where a region of the predetermined object exists inthe video; detecting the positions of the characteristic points in theframe in accordance with the obtained information; determining whetheror not the distance between the input position and the detected positionof the characteristic point is shorter than a reference value; anddetermining that the predetermined object has been specified when adetermination has been made that the distance is shorter than thereference value.

[0026] When a determination has been made that the predetermined objecthas been specified, it is desirable to show information related to thepredetermined object.

[0027] According to another aspect of the present invention, there isprovided a video processing method of displaying a region in which apredetermined object exists when the predetermined object has beenspecified in a screen which is displaying a video, the video processingmethod obtaining information describing parameter of a functionapproximating a trajectory obtained by arranging position data of atleast representative points of an approximate figure of the objectregion or characteristic points of the object region in a direction inwhich frames proceed when the region of the predetermined object existsin the video; detecting the representative point or the characteristicpoint in the frame in accordance with the obtained information; anddisplaying information for displaying the position of the object regionin the screen in a predetermined form of display in accordance with thedetected representative point or the characteristic point.

[0028] According to another aspect of the present invention, there isprovided a video processing method for retrieving a predetermined objectamong objects which appears in a video and which satisfies apredetermined condition, the video processing method inputting anarbitrary position in the video and a retrieving condition determined inaccordance with the input position; obtaining information describingparameter of a function approximating a trajectory obtained by arrangingposition data of representative points of an approximate figure of anobject region produced for each object which appears in the video or acharacteristic point of the object region in a direction in which framesproceed; determining, for each object over a plurality of frames,whether or not the representative point of the approximate figure or thecharacteristic point and the input position have a predeterminedrelationship in one frame of one object obtained in accordance with theobtained information; and detecting the predetermined object satisfyingthe retrieving condition in accordance with a result of determination.

[0029] The predetermined relationship may be the relationship that theinput position exists in the approximate figure region or therelationship that the distance from the characteristic point to theinput position is shorter than a reference value. The retrievingcondition may be a condition of an object which is to be extracted,which is selected from a retrieval condition group consisting of acondition that at least one frame satisfying the predeterminedrelationship exists at the input position, a condition that thepredetermined number of frames each satisfying the predeterminedrelationship exists successively with regard to the input position and acondition that the predetermined relationship is not satisfied in all ofthe frames.

[0030] The retrieval condition group includes, as a condition which mustbe added to the condition which is determined in accordance with theposition, an attribute condition which must be satisfied by theapproximate figure of the object.

[0031] According to another aspect of the present invention, there isprovided a video processing method for retrieving a predetermined objectamong objects which appears in a video and which satisfies apredetermined condition, the video processing method inputtinginformation for specifying a trajectory of the position in a video whichis to be retrieved; obtaining information describing parameter of afunction approximating a trajectory obtained by arranging position dataof representative points of an approximate figure of the object regionproduced for each object which appears in a video and which is to beretrieved or a characteristic point of the object region in a directionin which frames proceed; evaluating, for each object, similarity of thetrajectory of the representative point or the characteristic point ofthe one object detected in accordance with the obtained information andthe trajectory of the input position; and detecting the predeterminedobject corresponding to the specified trajectory.

[0032] Information for specifying the trajectory of the position may betime sequence information including the relationship between theposition and time. The similarity may be evaluated while the positionalrelationship is being added.

[0033] The specified trajectory may be a trajectory of an object in avideo which has been specified. Alternatively, a user may be permittedto input the trajectory by drawing the trajectory on a GUI.

[0034] According to another aspect of the present invention, there isprovided an object-region-data generating apparatus for generating dataabout described information of a region of an arbitrary object in avideo over a plurality of continuous frames, the object-region-datagenerating apparatus comprising a circuit configured to approximate anobject region in the video in a plurality of the subject frames by usinga predetermined figure; a detector configured to detect, in the pluralframes, coordinate values of the predetermined number of representativepoints identifying the predetermined figure which has been used in theapproximation; and a circuit configured to approximate a trajectory of atime sequence of the coordinate values of the representative pointsobtained over the plurality of the continuous frames with apredetermined function, so that information about the object region isgenerated by using the parameter of the function.

[0035] According to another aspect of the present invention, there isprovided an object-region-data generating apparatus for generating dataabout described information of a region of an arbitrary object in avideo over a plurality of continuous frames, the object-region-datagenerating apparatus comprising a detector configured to detect thecoordinate values of the predetermined number of characteristic pointsof an object region in a video over the plurality of the subject frames,and a circuit configured to approximate a time sequential trajectory ofthe coordinate values of the characteristic points obtained over theplurality of the continuous frames with a predetermined function,wherein the parameter of the function is used to generate informationabout the object region.

[0036] According to another aspect of the present invention, there isprovided a video processing apparatus for performing a predeterminedprocess when a predetermined object has been specified in a screen whichis displaying a video, the video processing apparatus comprising acircuit configured to obtain a parameter of a function approximating atrajectory obtained by arranging position data of representative pointsof an approximate figure of the object region in a direction in whichframes proceed in a case where a region of a predetermined object existsin the video when an arbitrary position has been specified in the screento detect the position of the representative point in the frame; adetector configured to detect the position of the approximate figure inaccordance with the detected position of the representative point; and acircuit configured to determine whether or not the input position existsin the approximate figure.

[0037] According to another aspect of the present invention, there isprovided a video processing apparatus for performing a predeterminedprocess when a predetermined object has been specified in a screen whichis displaying a video, the video processing apparatus comprising acircuit configured to obtain a parameter of a function approximating atrajectory obtained by arranging position data of a characteristic pointof the object region in a direction in which frames proceed in a casewhere the region of the predetermined object exists in the video whenarbitrary position has been specified in the screen to detect theposition of the characteristic point in the frame; and a circuitconfigured to determine whether or not the distance between the inputposition and the detected position of the characteristic point isshorter than a reference value.

[0038] According to another aspect of the present invention, there isprovided a video processing apparatus for performing a predeterminedprocess when a predetermined object has been is specified in a screenwhich is displaying a video, the video processing apparatus comprising acircuit configured to obtain a parameter of a function approximating atrajectory obtained by arranging position data of at least arepresentative point of an approximate figure of the object region or acharacteristic point of the object region in a direction in which framesproceed when the region of the predetermined object exists in the videoto detect the representative point or the characteristic point in theframe; and a circuit configured to display information for indicatingthe position of the object region in the screen in a predetermineddisplay form.

[0039] According to another aspect of the present invention, there isprovided a video processing apparatus for retrieving a predeterminedobject among objects which appears in a video and which satisfies anspecified condition, the video processing apparatus comprising a circuitconfigured to obtain information describing parameter of a functionapproximating a trajectory obtained by arranging position data ofrepresentative points of an approximate figure of the object regionproduced for each object which appears in a video which is to beretrieved or a characteristic point of the object region in a directionin which frames proceed when an arbitrary position in the video which isto be retrieved and a retrieving condition determined in accordance withthe position have been input; a circuit configured to determine, foreach object over a plurality of the frames, whether or not theapproximate figure or the characteristic point of one object in oneframe obtained in accordance with the obtained information and the inputposition satisfy a predetermined relationship; and a detector configuredto detect an object which satisfies the retrieving condition inaccordance with a result of the determination.

[0040] According to another aspect of the present invention, there isprovided a video processing apparatus for retrieving a predeterminedobject among objects which appears in a video and which satisfies anspecified condition, the video processing apparatus comprising a circuitconfigured to obtain information describing parameter of a functionapproximating a trajectory obtained by arranging position data ofrepresentative points of an approximate figure of the object regionproduced for each object which appears in the video which is to beretrieved or a characteristic point of the object region in a directionin which frames proceed when information for specifying a trajectory ofthe position in a video which is to be retrieved has been input; acircuit configured to evaluate, for each object, similarity between thetrajectory of the representative point or the characteristic point ofone object obtained in accordance with the obtained information and thetrajectory of the input position; and a detector configured to detectthe predetermined object corresponding to the specified trajectory inaccordance with the evaluated similarity.

[0041] Note that the present invention relating to the apparatus may beemployed as the method and the present invention relating to the methodmay be employed as the apparatus.

[0042] The present invention relating to the apparatus and the methodmay be employed as a recording medium which stores a program for causinga computer to perform the procedure according to the present invention(or causing the computer to serve as means corresponding to the presentinvention or causing the computer to realize the function correspondingto the present invention) and which can be read by the computer.

[0043] The present invention is configured such that the object regionin a video over a plurality of frames is described as a parameter of afunction approximating a trajectory obtained by arranging position dataof representative points of an approximate figure of the object regionor a characteristic point of the object region in a direction in whichframes proceed. Therefore, the object region in the video over theplural frames can be described with a small quantity of the functionparameters. Hence it follows that the quantity of data required toidentify the object region can effectively be reduced. Moreover,handling can be facilitated. Moreover, extraction of a representativepoint or a characteristic point from the approximate figure orgeneration of the parameter of the approximate curve can easily beperformed. Moreover, generation of an approximate figure from theparameter of the approximate curve can easily be performed.

[0044] When the representative point of the approximate figure isemployed, a fundamental figure, for example, one or more ellipses, areemployed such that each ellipse is represented by two focal points andanother point. Thus, whether or not arbitrary coordinates specified by auser exist in the object region (the approximate figure) can bedetermined by using a simple discriminant. Hence it follows that theuser is able to easily instruct a moving object in a video.

[0045] When the characteristic point is employed, whether or not thearbitrary coordinates specified by a user indicates the object regioncan considerably easily be determined. Thus, a moving object in a videocan easily be specified by the user.

[0046] When display of an object region among regions of objects whichcan be identified by using object region data and which has relatedinformation, or display of an image indicating the object region iscontrolled, the user is permitted to quickly recognize whether or notrelated information exists and the position of the object region.Therefore, the operation which is performed by the user can effectivelybe aided.

[0047] According to the present invention, retrieval of an object in avideo can easily be performed in accordance with a position in a videothrough which the object passes, residence time at a certain point or atrajectory.

[0048] Additional objects and advantages of the present invention willbe set forth in the description which follows, and in part will beobvious from the description, or may be learned by practice of thepresent invention.

[0049] The objects and advantages of the present invention may berealized and obtained by means of the instrumentalities and combinationsparticularly pointed out hereinafter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

[0050] The accompanying drawings, which are incorporated in andconstitute a part of the specification, illustrate presently preferredembodiments of the present invention and, together with the generaldescription given above and the detailed description of the preferredembodiments given below, serve to explain the principles of the presentinvention in which:

[0051]FIG. 1 is a diagram showing an example of the structure of anobject-region-data generating apparatus according to a first embodimentof the present invention;

[0052]FIGS. 2A, 2B, 2C and 2D are diagrams showing a procedure fordescribing an object region in a video with object region data;

[0053]FIG. 3 is a diagram showing an example of a process forapproximating an object region with an ellipse;

[0054]FIG. 4 is a diagram showing an example of a process for detectinga representative point of an approximate ellipse of an object region;

[0055]FIG. 5 is a diagram showing an example of the structure of objectregion data;

[0056]FIG. 6 is a diagram showing an example of the structure of data ofan approximate figure in object region data;

[0057]FIG. 7 is a diagram showing an example of the structure of data ofa trajectory of a representative point in data of an approximate figure;

[0058]FIG. 8 is a diagram showing an example of representative pointswhen the approximate figure is a parallelogram;

[0059]FIG. 9 is a diagram showing an example of representative pointswhen the approximate figure is a polygon;

[0060]FIG. 10 is a flowchart showing an example of a procedure accordingto the first embodiment of the present invention;

[0061]FIG. 11 is a diagram showing an example in which the object regionin a video is expressed with a plurality of ellipses;

[0062]FIG. 12 is a diagram showing an example of the structure of objectregion data including data of a plurality of approximate figures;

[0063]FIGS. 13A, 13B and 13C are diagrams schematically showing anotherprocess for describing an object region in a video with object regiondata;

[0064]FIG. 14 is a flowchart showing an example of a procedure forobtaining an approximate rectangle;

[0065]FIG. 15 is a diagram showing a state in which an inclined andelongated object is approximated with a non-inclined rectangle;

[0066]FIGS. 16A and 16B are diagrams showing a state in which an objecthas been approximated with a rectangle having an inclinationcorresponding to the inclination of the object;

[0067]FIG. 17 is a flowchart showing another example of a procedure forobtaining the approximate rectangle;

[0068]FIG. 18 is a diagram showing a method of obtaining an approximateellipse from an approximate rectangle;

[0069]FIG. 19 is a flowchart showing an example of a procedure forobtaining an approximate ellipse from an approximate rectangle;

[0070]FIG. 20 is a diagram showing a method of making representativepoints of approximate figures to correspond to one another betweenadjacent frames;

[0071]FIG. 21 is a flowchart showing an example of a procedure formaking representative points of approximate figures to correspond to oneanother between adjacent frames;

[0072]FIG. 22 is a diagram showing another example of the structure ofobject region data;

[0073]FIG. 23 is a diagram showing an example of the correspondenceamong the ID of types of figures, the type of the figures and the numberof representative points;

[0074]FIG. 24 is a diagram showing an example of the correspondenceamong the ID of a function, the form of the function and the functionparameter and the limit condition;

[0075]FIG. 25 is a diagram showing a specific example of the structureof data about related information;

[0076]FIG. 26 is a diagram showing another specific example of thestructure of data about related information;

[0077]FIG. 27 is a diagram showing an example of an object-region-datagenerating apparatus according to a second embodiment of the presentinvention;

[0078]FIG. 28 is a flowchart showing an example of a procedure accordingto the second embodiment;

[0079]FIG. 29 is a diagram showing an example of the structure of avideo processing apparatus according to a third embodiment of thepresent invention;

[0080]FIG. 30 is a flowchart showing an example of a procedure accordingto the third embodiment;

[0081]FIG. 31 is a diagram showing an example of display of contentshyper media which uses object region data;

[0082]FIG. 32 is a flowchart showing another example of the procedureaccording to the third embodiment;

[0083]FIG. 33 is a flowchart showing an example of a procedure accordingto a fourth embodiment of the present invention;

[0084]FIGS. 34A and 34B are diagrams showing an example of change in thedisplay of an object region having related information;

[0085]FIG. 35 is a diagram showing another example of the display of theposition of an object region having related information;

[0086]FIG. 36 is a diagram showing another example of the display of theposition of an object region having related information;

[0087]FIG. 37 is a diagram showing an example of display of adescription list of an object region having related information;

[0088]FIG. 38 is a diagram showing an example of display of an objectregion having related information with an icon;

[0089]FIG. 39 is a diagram of an example of display of an object regionhaving related information with a map;

[0090]FIGS. 40A and 40B are diagrams showing an example of control of animage reproducing rate for facilitating instruction of an object region;

[0091]FIG. 41 is a diagram showing an example which enables imagecapture for facilitating instruction of an object region;

[0092]FIG. 42 is a flowchart showing an example of a procedure accordingto a fifth embodiment of the present invention; and

[0093]FIG. 43 is a flowchart showing another example of the procedureaccording to the fifth embodiment.

DETAILED DESCRIPTION OF THE INVENTION

[0094] A preferred embodiment of an object-region-data generatingapparatus according to the present invention will now be described withreference to the accompanying drawings.

[0095] First Embodiment

[0096]FIG. 1 is a block diagram showing the structure of a firstembodiment of the present invention. As shown in FIG. 1, anobject-region-data generating apparatus incorporates a video datastorage portion 100, a region extracting portion 101, a region figureapproximating portion 102 for approximating a region with a figure, afigure-representative-point extracting portion 103, a representativepoint trajectory curve approximating portion 104 for approximatingrepresentative points with a curve, a related information storageportion 105 and a region data storage portion 106. A case will now bedescribed in which the process according to this embodiment (inparticular, processes arranged to be performed by the region extractingportion 101 or the region figure approximating portion 102) isconfigured such that the operation which is performed by a user ispermitted. In the foregoing case, the GUI (not shown in FIG. 1) isemployed with which video data is displayed in, for example, frame unitsto permit input of an instruction from the user.

[0097] The video data storage portion 100 stores video data andcomprises, for example, a hard disk, an optical disk or a semiconductormemory.

[0098] The region extracting portion 101 extracts a portion of regionsof video data. The portion of the regions are regions of an object, suchas a specific person, a vehicle or a building (as an alternative tothis, a portion of the object, for example, the head of a person, thebonnet of a vehicle or the front door of a building) in the video.Usually a video has the same object in the continuous frames thereof.The region corresponding to the same object frequently changes owing tothe movement of the object or shaking of a camera during an imagepick-up operation.

[0099] The region extracting portion 101 extracts an object region ineach frame corresponding to the movement or deformation of the object ofinterest. Specifically, the extraction is performed by a method ofmanually specifying a region in each of all of the frames. Anothermethod may be employed with which the contour of an object iscontinuously extracted by using an active contour model called “Snakes”as disclosed in “Snakes: Active contour models” (International Journalof Computer Vision, vol. 1, No. 4, pp. 321-331, July, 1988 disclosed byM. Kass et al.). Also a method disclosed in “Method of tracinghigh-speed mobile object for producing hyper media contents by usingrobust estimation” (CVIM 113-1, 1998, technical report of InformationProcessing Society of Japan) may be employed. According to thedisclosure, deformation and movement of the overall body of an objectare estimated in accordance with a position to which a partial objectregion has been moved and which has been detected by performing blockmatching. Alternatively, a method of identifying a region having similarcolors by performing growing and division of a region as disclosed inImage Analysis Handbook (Chapter-2, Section II, Publish Conference ofTokyo University, 1991) may be employed.

[0100] The region figure approximating portion 102 approximates anobject region in a video extracted by the region extracting portion 101with a predetermined figure. The figure may be an arbitrary figure, suchas a rectangle, a circle, an ellipse or a polygon. Also a method ofapproximating a region may be a method of performing approximation to afigure circumscribing the region. Another method of performingapproximation to a figure inscribing the region may be employed or amethod may be employed which is arranged such that the centroid of theregion is employed as the centroid of the approximate figure. Anothermethod of making the area ratio of the region and the approximate figureto be the same may be employed. As an alternative to the approximationof the object region with a predetermined type figure, the type of thefigure may be specified by a user for each object to which approximationis performed. Another method may be employed with which the type of thefigure is automatically selected in accordance with the shape of theobject or the like for each of the object to which approximation isperformed.

[0101] The approximation of the region with the figure is performed foreach frame whenever a result of extraction performed by the regionextracting portion 101 is input. Alternatively, approximation with afigure may be performed by using a result of extraction of a pluralityof preceding and following frames. When the result of extraction of theplural frame is employed, change in the size and position of theapproximate figure is smoothed among the plural frames so that themovement and deformation of the approximate figure are smoothed or anerror in the extraction of the region is made to be inconspicuous. Notethat the size of the approximate figure may vary among the frames.

[0102] The figure-representative-point extracting portion 103 extractsrepresentative points of the approximate figure which is an output ofthe region figure approximating portion 102. The point which is employedas the representative point varies according to the type of the employedapproximate figure. When the approximate figure is formed into, forexample, rectangle, the four or three vertices of the rectangle may bethe representative points. When the approximate figure is formed into acircle, the representative points may be the center and one point on thecircumference or two end points of the diameter. When the approximatefigure is an ellipse, the representative points may be the vertex of acircumscribed rectangle of the ellipse or the two focal points and onepoint on the ellipse (for example, one point on the minor axis). When anarbitrary closed polygon is the approximate figure, the vertices may bethe representative points of the figure.

[0103] The representative points are extracted in frame units wheneverinformation about the approximate figure for one frame is output fromthe region figure approximating portion 102. Each representative pointis expressed by the coordinate axis in the horizontal (X) direction andthe coordinate axis in the vertical (Y) direction.

[0104] The representative point trajectory curve approximating portion104 time-sequentially approximates the sequence of the representativepoints extracted by the figure-representative-point extracting portion103 to a curve. The approximate curve is, for each of the X coordinateand Y coordinate of each representative point, expressed as a functionof the frame number f or time stamp t given to the video. Theapproximation with the curve may be approximation with a straight lineor approximation with a spline curve.

[0105] The related information storage portion 105 stores information(as an alternative to this, information about the address at whichrelated information stored in another storage apparatus, for example,Internet or a server on a LAN) relating to the object which appears invideo data stored in the video data storage portion 100. Relatedinformation may be a character, voice, a still image, a moving image ortheir combination. Alternatively, related information may be datadescribing the operation of a program or a computer. Similarly to thevideo data storage portion 100, the related information storage portion105 comprises a hard disk, an optical disk or a semiconductor memory.

[0106] The region data storage portion 106 is a storage medium in whichobject region data is stored which includes data for expressing aformula of the curve approximating the time-sequential trajectory of therepresentative points which is the output of the representative pointtrajectory curve approximating portion 104. When related informationabout the object corresponding to the region expressed by a function hasbeen stored in the related information storage portion 105, objectregion data may include related information and the address at whichrelated information has been recorded. When information of the addressof recorded related information has been stored in the relatedinformation storage portion 105, also address information may berecorded. Similarly to the video data storage portion 100 and therelated information storage portion 105, the region data storage portion106 comprises a hard disk, an optical disk or a semiconductor memory.

[0107] The video data storage portion 100, the related informationstorage portion 105 and the region data storage portion 106 may beconstituted by individual pieces of storage apparatus. Alternatively,the overall portion or a portion may be constituted by one storageapparatus.

[0108] The object-region-data generating apparatus may be realized by asoftware which is operated on a computer.

[0109] The operation of the object-region-data generating apparatus willspecifically be described.

[0110]FIGS. 2A, 2B, 2C and 2D are diagrams more specifically showing asequential process. The sequential process includes a process which isperformed by the region extracting portion 101 to extract the objectregion. Moreover, a process which is performed by the region figureapproximating portion 102 to approximate the region and a process whichis performed by the figure-representative-point extracting portion 103to extract a representative point of a figure are included. Also aprocess which is performed by the representative point trajectory curveapproximating portion 104 to approximate the representative pointtrajectory with a curve is included.

[0111] In this case, the region figure approximating portion 102 employsa method of approximating the region with an ellipse. Thefigure-representative-point extracting portion 103 employs a method ofextracting the two focal points of the ellipse and one point on theellipse. The representative point trajectory curve approximating portion104 employs a method of approximating the sequence of the representativepoints with a spline function.

[0112] Referring to FIG. 2A, reference numeral 200 represents a video ofone frame which is to be processed. Reference numeral 201 represents theobject region which is to be extracted. A process for extracting theobject region 201 is performed by the region extracting portion 101.Reference numeral 202 represents an ellipse which is a result ofapproximation of the object region 201 with an ellipse. A process forobtaining the ellipse 202 from the object region 201 is performed by theregion figure approximating portion 102.

[0113]FIG. 3 shows an example of the method of obtaining an approximateellipse when the object region is expressed by a parallelogram. PointsA, B, C and D shown in FIG. 3 are vertices of the parallelogram which isthe object region. In the foregoing case, calculations are performed sothat which side AB or side BC is a longer side is determined. Then, asmallest rectangle having portions of its sides which are the longerside and its opposite side is determined. In the case shown in FIG. 3, arectangle having four points A, B′, C and D′ is the smallest rectangle.The approximate ellipse is a circumscribing ellipse similar to theellipse inscribing the rectangle and passing the points A, B′, C and D′.

[0114] Referring to FIG. 2B, reference numerals 203 representrepresentative points of a figure expressing an ellipse. Specifically,the representative points are two focal points of the ellipse and onepoint on the same (one point on the minor axis in the case shown in FIG.2B). The focal points of the ellipse can easily be determined frompoints on the two axes or a circumscribing rectangle of the ellipse. Anexample will now be described with which focal points F and G aredetermined from two points P₀ and P₁ on the major axis and point H onthe minor axis shown FIG. 4.

[0115] Initially, a and b which are parameters of the major axis and theminor axis, center C of the ellipse and eccentricity e are determined asfollows:

E(P ₀ , P ₁)=2×a

C=(P ₀ +P ₁)/2

E(C, H)=b

e=(1/a)×{square root}{square root over ((a×a−b×b))}

[0116] where E (P, Q) is the Euclidean distance between the point P andthe point Q. In accordance with the determined parameters, the focalpoints F and G can be determined as follows:

F=C+e×(P ₀ −C)

G=C−e×(P ₀ −C)

[0117] Thus, the representative points F, G and H of the ellipse aredetermined. When the foregoing points are made to correspond to therepresentative points of the ellipse extracted in another frame,ambiguity is involved. That is, two combinations exist which make thetwo extracted focal points correspond to the two focal points in theprevious frame. Since two intersections exist between the minor axis andthe ellipse, the intersection corresponding to the one point on theellipse extracted in the previous frame cannot be determined. A methodof determining the combination and the intersection will now bedescribed.

[0118] An assumption is made that the two focal points extracted in theprevious frame are Fp and Gp. To determine F or G which correspond toFp, the following comparison is made:

E((Gp−Fp)/2, (G−F)/2) and

E((Gp−Fp)/2, (F−G)/2)

[0119] When the former focal point is smaller, Fp is made to correspondto F, and Gp is made to correspond to G. When the latter focal point issmaller, Fp is made to correspond to G and, Gp is made to correspond toF.

[0120] An assumption is made that the intersections between the minoraxis and the ellipse in the previous frame are Hp and the intersectionsbetween the minor axis of the ellipse in the present frame are H and H′.The point H or H′ which must be made to correspond to Hp is determinedby calculating two distances:

E(Hp−(Gp+Fp)/2, H−(F+G)/2) and

E(Hp−(Gp+Fp)/2, H′−(F+G)/2)

[0121] When the former distance is shorter, H is selected. In a negativecase, H′ is selected. Note that the intersection H between the minoraxis and the ellipse in the first frame may be either of the twointersections.

[0122] The foregoing process for extracting the representative pointsfrom the ellipse is performed by the figure-representative-pointextracting portion 103.

[0123] The representative points extracted by the foregoing process areusually varied in the position among the successive frames owing tomovement of the object of interest in the video or shaking of the imagepick-up camera. Therefore, the corresponding representative points ofthe ellipses are time-sequentially arranged to perform approximationwith a spline function for each of the X and Y axes. In this embodiment,each of the three points F, G and H (see FIG. 4) which are therepresentative points of the ellipse requires a spline function for theX and Y coordinates. Therefore, six spline functions are produced.

[0124] The approximation to a curve with spline functions is performedby the representative point trajectory curve approximating portion 104.

[0125] The process which is performed by the representative pointtrajectory curve approximating portion 104 may be carried out wheneverthe coordinates of the representative points of each frame relating tothe object region are obtained. For example, the approximation isperformed whenever the coordinates of the representative points in eachframe are obtained. Moreover, an approximation error is obtained toarbitrarily divide the approximation region in such a manner that theapproximation error satisfies a predetermined range. Another method maybe employed with which the process is performed after the coordinates ofthe representative points in all of the frames relating to the objectregion have been obtained.

[0126] Reference numeral 204 shown in FIG. 2C represents theapproximated spline function expressed three-dimensionally. Referencenumeral 205 shown in FIG. 2D represents an example of the splinefunction which is the output of the representative point trajectorycurve approximating portion 104 (only one axis of coordinate of onerepresentative point is shown). In this example, the approximationregion is divided into two sections (the number of knots is three) whichare t=0 to 5 and t=5 to 16.

[0127] The thus-obtained spline functions are recorded in the regiondata storage portion 106 in a predetermined data format.

[0128] As described above, this embodiment enables the object region ina video to be described as the parameter of a curve approximating atime-sequential trajectory (a trajectory of the coordinates of therepresentative points having the variable are the frame numbers or thetime stamps) of the representative points of the approximate figure ofthe region.

[0129] According to this embodiment, the object region in a video can beexpressed by only the parameters of the function. Therefore, objectregion data, the quantity of which is small and which can easily behandled, can be produced. Also extraction of representative points fromthe approximate figure and producing of parameters of the approximatecurve can easily be performed. Moreover, producing of an approximatefigure from the parameters of the approximate curve can easily beperformed.

[0130] A method may be employed with which a basic figure, for example,one or more ellipses are employed as the approximate figures and eachellipse is represented by two focal points and another point. In theforegoing case, whether or not arbitrary coordinates specified by a userexist in the region (the approximate figure) of the object (whether ornot the object region has been specified) can be determined by a simpledeterminant. Thus, specification of a moving object in a video canfurthermore easily be performed by the user.

[0131] The data format of object region data which is stored in theregion data storage portion 106 will now be described. A case will nowbe described in which the representative points are approximated with aspline function. As a matter of course, a case in which therepresentative points are approximated with another function isperformed similarly.

[0132]FIG. 5 shows an example of the data format of object region datafor recording the spline function indicating the object region in avideo and information related to the object.

[0133] ID number 400 is an identification number which is given to eachobject. Note that the foregoing data item may be omitted.

[0134] A leading frame number 401 and a trailing frame number 402 areleading and trailing frame numbers for defining existence of the objecthaving the foregoing ID number. Specifically, the numbers 401 and 402are the frame number at which the object appears in the video and theframe number at which the object disappears. The frame numbers are notrequired to be the frame numbers at which the object actually appearsand disappears in the video. For example, an arbitrary frame numberafter the appearance of the object in the video may be the leading framenumber. An arbitrary frame number which follows the leading frame numberand which precedes the frame of disappearance of the object in the videomay be the trailing frame number. The leading/trailing time stamp may besubstituted for the lading/trailing frame number. The object existenceframe number or object existence time may be substituted for thetrailing frame number 402.

[0135] A pointer (hereinafter called a “related information pointer”)403 for pointing related information is the address or the like of thedata region in which data of information related to the object havingthe foregoing ID number. When the related information pointer 403 forpointing related information is used, retrieval and display ofinformation related to the object can easily be performed. The relatedinformation pointer 403 for pointing related information may be pointerfor pointing data of description of a program or the operation of acomputer. In the foregoing case, when the object has been specified by auser, the computer performs a predetermined operation.

[0136] Note that the related information pointer 403 for pointingrelated information may be omitted when the objects are not required toperform individual operations.

[0137] The operation for describing the related information pointer 403for pointing related information in the object region data will now bedescribed. As an alternative to using the pointer 403, relatedinformation itself may be described in object region data. The relatedinformation pointer 403 for pointing related information and relatedinformation may be described in object region data. In the foregoingcase, a flag is required to indicate whether the related informationpointer for pointing related information or related information has beendescribed in object region data.

[0138] The approximate figure number 404 is the number of the figuresapproximating the object region. In the example shown in FIG. 2A inwhich the object region is approximated with one ellipse, the number ofthe figures is 1.

[0139] Approximate figure data 405 is data (for example, the parameterof a spline function) of a trajectory of the representative point of thefigure for expressing an approximate figure.

[0140] Note that approximate figure data 405 exists by the numbercorresponding to the approximate figure number 404 (a case where theapproximate figure number 404 is two or larger will be described later).

[0141] The number of the approximate figure number 404 for object regiondata may always be one (therefore, also approximate figure data 405 isalways one) to omit the field for the approximate figure number 404.

[0142]FIG. 6 shows the structure of approximate figure data 405 (seeFIG. 5).

[0143] A figure type ID 1300 is identification data for indicating thetype of a figure serving as the approximate figure, the figure type ID1300 being arranged to identify a circle, an ellipse, a rectangle and apolygon.

[0144] A representative point number 1301 indicates the number ofrepresentative points of the figure specified by the figure type ID1300. Note that the number of the representative points is expressedwith M.

[0145] A pair of representative point trajectory data items 1302 and1303 are data regions relating to the spline function for expressing thetrajectory of the representative points of the figure. Therepresentative points of one figure require data of one pair of splinefunctions for the X and Y coordinates. Therefore, data of the trajectoryof the representative points for specifying the spline function existsby representative point number (M)×2.

[0146] Note that the type of the employed approximate figure maypreviously be limited to one type, for example, an ellipse. In theforegoing case, the field for the figure type ID 1300 shown in FIG. 6may be omitted.

[0147] When the representative point number is defined according to thefigure type ID 1300, the representative point number may be omitted.

[0148]FIG. 7 shows an example of the structure of representative pointtrajectory data 1302 and 1303.

[0149] A knot frame number 1400 indicates the knots of the splinefunction. Thus, a fact that polynomial data 1403 is effective to theknots is indicated. The number of coefficient data 1402 of thepolynomial varies according to the highest order of the spline function(assuming that the highest order is K, the number of coefficient data isK+1). Therefore, reference to a polynomial order 1401 is made.Subsequent to the polynomial order 1401, polynomial coefficients 1402 bythe number corresponding to the polynomial order (K)+1 follows.

[0150] Since the spline function is expressed in an individualpolynomial among the knots, the polynomials are required by the numbercorresponding to the number of knots. Therefore, data 1403 including theknot frame number and the coefficient of the polynomial is describedrepeatedly. When the knot frame number is the same as the trailing endframe, it means the trailing end polynomial coefficient data. Therefore,termination of representative point trajectory data can be understood.

[0151] A case will now be described in which a figure except for theellipse is employed as the approximate figure.

[0152]FIG. 8 is diagram showing the representative points in a casewhere a parallelogram is employed as the approximate figure. Points, A,B, C and D are vertices of the parallelogram. Since three points of thefour vertices are determined, the residual one is determined. Therefore,three vertices among the four vertices are required to serve as therepresentative points. In the foregoing example, three points, which areA, B and C, are employed as the representative points.

[0153]FIG. 9 is a diagram showing representative points in a case wherea polygon is employed to serve as the approximate figure. In the case ofthe polygon, the order of the vertices is made to be the order along theouter surface. Since the example shown in FIG. 9 has 10 vertices, all ofthe vertices N₁ to N₁₀ are employed as the representative points. In theforegoing case, the number of the vertices may be reduced by employingonly vertices each having an internal angle smaller than 180° as therepresentative points.

[0154] As described above, the foregoing process may be performed bysoftware which is operated on a computer. FIG. 10 is a flowchart showingthe process which is performed by the video processing apparatusaccording to this embodiment. When the video processing apparatusaccording to this embodiment is realized by software, a programaccording to the flowchart shown in FIG. 10 is produced.

[0155] In step S11, video data for one frame is extracted from the videodata storage portion 100.

[0156] In step S12, the region of a predetermined object in the video isextracted. Extraction may be performed by a method similar to thatemployed by the region extracting portion 101.

[0157] In step S13, an approximate figure is approximated to region datawhich is a result of the process performed in step S12. Theapproximation method may be similar to that employed by the regionfigure approximating portion 102.

[0158] In step S14, the representative points of the figure approximatedin step S13 is extracted. Also a method similar to that employed by thefigure-representative-point extracting portion 103 may be employed.

[0159] In step S15, approximation of the position of a representativepoint train of the approximate figure in the successive frame with acurve is performed. Also a method similar to that employed by therepresentative point trajectory curve approximating portion 104 may beemployed.

[0160] In step S16, a branching process is performed. Thus,determination is made whether or not the processed image is in the finalframe or whether or not the object in the processed frequency which isto be extracted has disappeared from the image (or considered that theobject has disappeared). In an affirmative case, the process is branchedto step S17. In a negative case (both of the cases are negated), theprocess is branched to step S11.

[0161] In step S17, the approximate curve calculated in step S15 isrecorded in a recording medium as object region data in accordance witha predetermined format.

[0162] The example has been described with which one figure is assignedto one object to roughly express the object region. The accuracy ofapproximation may be improved by making approximation to the region ofone object with a plurality of figures. FIG. 11 shows an example inwhich a plurality of figures are approximated to one object. In theforegoing case, a region of a person in the image is expressed with 6ellipses 600 to 605.

[0163] When one object is expressed with the plural figures as shown inFIG. 11, a process for dividing the object into a plurality of regionsmust be performed. The process may be performed by an arbitrary method.For example, a method with which the object is directly divided withmanpower may be employed. In the foregoing case, a pointing device, suchas a mouse, is used to, on the image, enclose the region with arectangle or an ellipse. Alternatively, the region is specified with atrajectory of the pointing device. When an automatic method is employedas a substitute for the manpower, a method may be employed with whichclustering of movement of the object is performed to realize thedivision. The foregoing method is a method with which the movement ofeach region in the object among the successive frames is determined by acorrelation method (refer to, for example, Image Analysis HandbookChapter-3, Section II, Publish Conference of Tokyo University, 1991) ora method with gradient constraints (refer to, for example, Determiningoptical flow, B. K. P. Horn and B. G. Schunck, Artificial Intelligence,Vol. 17, pp. 185-203, 1981) to collect similar movements to form aregion.

[0164] Each of the divided regions is subjected to the process which isperformed by the example of the structure shown in FIG. 1 or theprocedure shown in FIG. 10 so that data of the approximate figure isproduced. In the foregoing case, the spline function, which must bedescribed in object region data of one object increases as the number ofthe approximate figures increases. Therefore, the structure of data isformed which includes approximate figure data 405 by the number (L inthe foregoing case) corresponding to the approximate figure number 404,as shown in FIG. 12.

[0165] As described above, the field for the approximate figure number404 may be omitted by making the approximate figure number to always beone (therefore, data of the approximate figure is made to always be one)to the object region data. In the foregoing case, one object can beexpressed with a plurality of figures when object region data isproduced for each figure approximating one object (the same ID number isgiven). That is, approximate figure data (l) to approximate figure data(L) 405 shown in FIG. 12 is required to be substituted for partial data(l) to partial data (L) in a certain region (for example, a region 605).

[0166] When one object is expressed with a plurality of figures in thisembodiment, the same figure is employed. A mixture of a plurality typesof figures may be employed.

[0167] Variation of a method of use of region data produced and recordedin this embodiment will now be described. Although a person, an animal,a building or a plant is considered as the object in a video, theprocess according to this embodiment may be applied to any object in thevideo. For example, a telop may be handled as an object in a video.Therefore, a process in which a telop is employed as the variations ofthe object will now be described.

[0168] The telop is character information added to the image. In U.S.character information called a “closed caption” must be added. Inbroadcasts in Japan frequencies of use of telops have been increased.The telop which must be displayed includes a moving telop, such as astill telop, a telop which is scrolled upwards in the screen and a telopwhich is scrolled from right to the left of the screen. When the regionin which the telop is being displayed is approximated with a figure tostore the telop character train as related information, the contents ofthe image can easily be recognized or a predetermined image can easilybe retrieved.

[0169] The region extracting portion 101 performs a process by employinga method with which a telop region is manually specified. Another methodmay be employed which has been disclosed in “Method of ExtractingCharacter Portion from Video to Recognize Telop” (Hori, 99-CV1M-114, pp.129-136, 1999, “Information Processing Society of Japan TechnicalReport”) and with which the brightness and edge information ofcharacters are employed to perform character train extracting method.Another method has been disclosed in “Improvement in Accuracy ofNewspaper Story Based on Telop Character Recognition of News Video”(Katayama et al. Vol. 1, pp. 105-110, proceedings of Meeting on Image(Recognition and Understanding (MIRU '98)) to separate background andthe telop from each other by examining the intensity of edges. Thus, thetelop region is extracted. Each character and each character train maybe cut from the obtained telop region. Edge information in the telopregion in successive frames is compared with each other to detect aframe in which the telop has appeared and a frame in which the same hasdisappeared.

[0170] The region figure approximating portion 102 performs a process toapproximate the telop region extracted by the region extracting portion101 with a rectangle. The number of the frequency in which the telop hasappeared is stored in the leading frame number of object region data(401 shown in FIG. 5 or FIG. 12). On the other hand, the frame in whichthe telop has disappeared is stored in the trailing frame number 402. Apointer for pointing the character train information of the telop isstored in the related information pointer 403 for pointing relatedinformation. As approximate figure data 405, rectangular region dataencircling the telop is stored. When each row of a telop composed of aplurality of rows is made to be an individual region or when eachcharacter is made to be an individual region, the number of rows orcharacters is stored in the approximate figure number 404. Rectangularregion data encircling each row or character, that is, approximatefigure data 405, is stored by the corresponding number.

[0171] The figure-representative-point extracting portion 103 and therepresentative point trajectory curve approximating portion 104 performprocesses as described above because any specialized portion for thetelop is included in the processes.

[0172] The character train information of the telop which has appearedis stored in the related information storage portion 105. Moreover, thepointer for pointing information above is stored in telop region data(object region data).

[0173] When a keyword has been input and a character train correspondingor relating to the keyword is included in the character traininformation of the telop, the frame and time at which the charactertrain appears can easily be detected. If the image is a news program,retrieval of interesting articles is permitted to look only thearticles.

[0174] In the foregoing case, addition of a pointer for pointing objectregion data corresponding to the frame or time to the character traininformation of the telop facilitates the retrieval.

[0175] Thus, the telop is processed as described above. Variations ofthe object may be applied to the method of using this embodiment.

[0176] Although the method of approximation using the ellipse has beendescribed in the structure shown in FIG. 2, an approximation methodusing a rectangle will now be described as another approximation method.

[0177]FIGS. 13A, 13B and 13C are diagrams formed into the same shape asthat of FIGS. 2A, 2B, 2C and 2D. In the foregoing case, the regionfigure approximating portion 102 employs a method of approximating aregion with a rectangle. The figure-representative-point extractingportion 103 employs a method of extracting the four vertices of therectangle. The representative point trajectory curve approximatingportion 104 employs an approximation method using a spline function.

[0178] Referring to FIG. 13A, reference numeral 2800 represents videodata for one frame which is to be processed.

[0179] Reference numeral 2801 represents an object region which is to beextracted. A process for extracting the region 2801 of the object isperformed by the region extracting portion 101.

[0180] Reference numeral 2802 represents a result of approximation ofthe object region with the rectangle. The process for obtaining therectangle 2802 from the object region 2801 is performed by the regionfigure approximating portion 102.

[0181] An example of the process for obtaining the rectangle 2802 shownin FIG. 13A is shown in FIG. 14. That is, a mask image of the frame 2800is raster-scanned (step S60). When the subject pixel is included in theobject region (step S61), the minimum value is updated if each of the Xand Y coordinates is smaller than the stored minimum value. If thevalues are larger than the maximum value, the maximum value is updated(step S62). The foregoing process is repeated and checked for all of thepixels so that the minimum and maximum values of the pixel positionindicating the object region 2801 for each of the X and Y coordinatesare obtained. Thus, the coordinates of the four vertices of therectangle 2802 can be obtained.

[0182] Although the above-mentioned method is excellent in easiness ofthe process, a multiplicity of non-object regions are undesirablycontained in the approximate rectangle 3002 when, for example, as shownin FIG. 15, an elongated object 3001 exists diagonally with respect to ascreen 3000. When the elongated object is rotated, the size and shape ofthe rectangle 2802 are changed. The foregoing facts sometimes obstructidentification and instruction of the object.

[0183] An example of the approximation method will now be described withwhich the size of the rectangle can be minimized (the number of thenon-object regions in the approximate rectangle can be minimized) and towhich the attitude of the object can be reflected.

[0184] Referring to FIG. 16A, reference numeral 3100 represents a videofor one frame which is to be processed.

[0185] Reference numeral 3101 represents an object region which is to beextracted. A process for extracting the object region 3101 is performedby the region extracting portion 101.

[0186] Reference numeral 3102 represents a result of approximation ofthe object region. As distinct from the rectangle 2802 shown in FIG.13A, the foregoing approximate rectangle 3102 is inclined. Also only asmall number of the non-object regions exists in the region 3102. Whenthe subject has been rotated, the shape of the region 3102 is notchanged. The process for obtaining the rectangle 3102 from the objectregion 3101 is performed by the region figure approximating portion 102.

[0187]FIG. 17 shows an example of the process. The process is arrangedsuch that a principal axis of inertia of the object region is obtained.Moreover, an approximate figure is obtained in accordance with theprincipal axis of inertia.

[0188] Referring to FIG. 16B, reference numeral 3103 represents thecentroid of the object region 3101.

[0189] Reference numeral 3104 represents the principal axis of inertiaof the object region 3101. Reference numeral 3105 represents a straightline perpendicular to the centroid 3104.

[0190] Initially, inertia moments m₂₀, m₀₂ and m₁₁ of the object regionare obtained (steps S70 to S72).

[0191] Assuming that the mask image is f(x, y), f(x, y) is 1 in theregion 3101 and 0 on the outside of the region 3101. The inertia momentof the subject 3101 can be expressed as follows:

m _(ij) =ΣΣx ^(i) y ^(j) f(x, y)

[0192] The inertia moment of f(x, y) with respect to a straight line y=xtan θ passing through the origin is obtained as follows:

m _(θ)=∫∫(x sin θ−y cos θ)² f(x, y)dxdy

[0193] An assumption is made that the angle with which m_(θ) isminimized when θ has been changed is θ₀. When only one set of anglesexists, the straight line y=x tan θ₀ is called the principal axis ofinertia.

[0194] Note that tan θ₀ can be obtained as a solution of the followingquadratic equation:

tan²θ+{(m ₂₀ −m ₀₂)/m ₁₁} tan θ−1=0

[0195] When tan θ₀ is obtained around the centroid 3103, the relatedinformation of the object can be obtained (step S73).

[0196] Then, a straight line in parallel with the principal axis ofinertia and circumscribing the object region and a straight lineperpendicular to the principal axis of inertia and circumscribing theobject region are obtained (step S74).

[0197] Referring to FIG. 16B, straight lines 3106 and 3107 are inparallel with the principal axis of inertia 3104. The straight lines3106 and 3107 circumscribes the object region.

[0198] Straight lines 3108 and 3109 are straight lines in parallel withthe straight line 3105, the straight lines 3108 and 3109 circumscribingthe object region.

[0199] The rectangle 3102 is formed by the straight lines 3106, 3107,3108 and 3109 (step S75).

[0200] When the object is formed into a circle, the principal axis ofinertia cannot be obtained. In the foregoing case, a procedure, forexample, as shown in FIG. 14, may be employed to obtain an approximaterectangle.

[0201] The object region can sometimes more satisfactorily be expressedby an ellipse as compared with expression by the rectangle. FIG. 18shows an example of a method of an approximate ellipse from a rectanglewhen the object region is expressed with the rectangle. FIG. 19 shows anexample of a process employed in the foregoing case.

[0202] Referring to FIG. 18, an assumption is made that an object region3300 and a circumscribing rectangle 3301 have been obtained.

[0203] Initially, the inscribing ellipse and the circumscribing ellipseof the approximate rectangle 3301 are obtained (step S80).

[0204] Referring to FIG. 18, an ellipse 3302 is an inscribing ellipse ofthe rectangle 3301 and the ellipse 3303 is an circumscribing ellipse ofthe rectangle 3301.

[0205] Then, the size of the inscribing ellipse 3302 is graduallybrought closer to that of the circumscribing ellipse 3303 (step S81).Then, an ellipse 3304 for completely including the object region 3300 isobtained (step S82) to employ the ellipse 3304 as the approximateellipse. The unit for enlarging the size of the inscribing ellipse 3302in each process of the repeated process may previously be determined.The unit may be determined in accordance with the difference between thesize of the inscribing ellipse 3302 and that of the circumscribingellipse 3303.

[0206] A reverse method may be employed with which the size of thecircumscribing ellipse 3303 is brought closer to the size of theinscribing ellipse 3302. In the foregoing case, the circumscribingellipse 3303 includes the object region 3300 from the first. Therefore,the ellipse previous to the ellipse with which the portion which is notincluded in the object region 3300 has first occurred in the repeatedprocess is required to be the approximate ellipse 3304.

[0207] Then, the figure-representative-point extracting portion 103obtains the representative points of the approximate rectangle or theapproximate ellipse. The representative points of a rectangle may be thefour or three vertices of the rectangle. The representative points ofthe ellipse may be the vertices of the circumscribing rectangle or twofocal points and one point on the ellipse.

[0208] Then, the representative point trajectory curve approximatingportion 104 approximates the trajectory of the representative pointsobtained in the time sequential manner with a spline function or thelike. At this time, it is important to bring the time sequences intocorrespondence with each other. When the approximate figure is in theform of a rectangle and having the representative points which are thevertices, the vertices of the adjacent frames must be brought intocorrespondence with each other.

[0209]FIG. 20 shows an example of a method of a correspondence process.FIG. 21 shows an example of the procedure of the correspondence process.

[0210] Referring to FIG. 20, reference numeral 3500 represents thecentroid of the approximate rectangle. A rectangle 3501 in the previousframe and a rectangle 3502 in the present frame have been obtained.Either of the rectangle 3501 or 3502 is moved in parallel to make thecentroids to coincide with each other (a state in which the centroidshave been made coincide with each other is shown in FIG. 20). Distancesd1 to d4 between the vertices of the two rectangles are calculated toobtain the sum of the distances in the combinations of all of thevertices (steps S90 and S91). A combination with which the sum of thedistances made to be shortest is detected to establish thecorrespondence (step S92).

[0211] When representative points are obtained from the approximatefigure, the number of combinations which is obtained in step S91 can bereduced when the representative points are obtained by a predeterminedrule. When the coordinates of the vertices of a rectangle are storedclockwise, only four combinations is required for the correspondence.

[0212] The foregoing method sometimes has difficulty in realizing thecorresponding state. When the approximate rectangle is formed into asquare-like shape between the adjacent frames and the approximaterectangle has been rotated by 45°, the corresponding state cannot easilybe realized (because the sums of the distances are made to be similarvalues between the two combinations). In the foregoing case, a methodmay be employed with which the exclusive OR is obtained between theregions of the object in the approximate rectangle to employ acombination with which the area is minimized. Another method may beemployed with which an absolute difference between textures of theobject region is obtained to detect a combination with which thedifference is minimized.

[0213] An example will now be described in which when a trajectory ofthe object region is described by the method according to the presentinvention, the structure of data which is different from the approximatedata structure shown in FIGS. 6 and 7 is employed.

[0214]FIG. 22 shows another example of a description format for data ofthe approximate figure and data of trajectories of representative pointsof the object region. Note that FIG. 22 shows only one representativepoint for a portion (portion from knot number (N) 3902 to a functionspecifying information arrangement 3913) of data of the trajectory ofthe representative point (in actual, a plurality of representativepoints are described to correspond to the number of the representativepoints).

[0215] Description will now be made on the assumption that the highestorder of the polynomial is the second order.

[0216] In the foregoing example (shown in FIGS. 5, 6 and 7), all of thecoefficients of the polynomial spline function are described. Thedescription method in this example is arranged to use combination of thecoordinate of the knot of the spline function and a value relating tothe second-order coefficient of the spline function. The foregoingdescription method has an advantage that the knot can easily beextracted to cause the trajectory of a large object to easily bedetected.

[0217] The foregoing description method will now be described.

[0218] The figure type ID 3900 shown in FIG. 22 specifies the type ofthe figure which has been used to make the approximation of the shape ofan object. For example, only the centroid, the rectangle, the ellipse ortheir combination can be specified. FIG. 23 shows an example of types ofthe figures and assignment of the figure type ID. A representative pointnumber 3901 indicates the number of the trajectories of therepresentative points which are determined in accordance with the typeof the figure.

[0219] The knot number (N) 3902 indicates the number of knots of aspline function expressing the trajectory of the representative point.The frame corresponding to each knot is expressed as time so as to bestored in knot time (1) to knot time (N) 3903. Since a predeterminednumber of knot time has been provided, the knot time is described asknot time arrangement 3904.

[0220] Also x and y coordinates of each knot are described asarrangements 3906 and 3908 of X coordinate 3905 of the knot and the Ycoordinate 3907 of the knot.

[0221] A linear function flag 3909 indicates whether or not only alinear function is employed as the spline function between knots. Ifsecond or higher order polynomial is partially employed, the foregoingflag 3909 is turned off. Since the foregoing flag 3909 is employed,description of function specifying information 3910 to be describedlater which is employed when only the linear function is employed as theapproximate function can be omitted. Therefore, an advantage can berealized in that the quantity of data can be reduced. Note that the flagmay be omitted.

[0222] A function ID 3911 and a function parameter 3912 contained infunction specifying information 3910 indicate the order of thepolynomial spline function and information for specifying thecoefficient of the polynomial spline function, respectively. FIG. 24shows their examples. Note that ta and tb are time of continuous knots,f(t) is a spline function in a region [ta, tb] and, fa and fb arecoordinates of the knot at time ta and tb. Since information about theknot is sufficient information when a liner polynomial is employed, nofunction parameter is described. When a quadratic polynomial isemployed, one value is described in the function parameter asinformation for identifying the coefficient. Although the quadraticcoefficient is employed in the example shown in FIG. 24, another value,for example, one point on the quadratic curve except for fa and fb maybe employed.

[0223] The foregoing description method is able to regenerate the splinefunction in all regions in accordance with information about the knotsand the function parameter under the limitation conditions shown in FIG.24.

[0224] Function specifying information 3910 exists by the numbercorresponding to knot number N−1, the function specifying information3910 being described as an arrangement 3913.

[0225] Although the description has been made that the highest order ofthe polynomial is the quadratic order, the highest order of thepolynomial may, of course, be a cubic or higher order.

[0226] The variations of related information will now be described.

[0227]FIG. 25 shows an example of the structure of data 4200 aboutrelated information for use in a monitor video. Actual data is requiredto contain at least one item.

[0228] An object type 4201 is data indicating the type, such as a“vehicle” or a “person”, of an object to which approximation is made.

[0229] Identification information 4202 is data for identifying an actualobject, such as “name of a person”, “the license number of a vehicle” or“the type of the vehicle”.

[0230] An operation content 4203 is data indicating the operation, suchas “walking” or “running” of the object.

[0231]FIG. 26 shows an example of the structure of data 4300 aboutrelated information for mainly use in a commercial contents or hypermedia contents. Actual data is required to contain at least one item.

[0232] Name 4301 is data indicating name of the object. In a case wherethe object is a character of a movie or the like, name of the player orthe actor is specified.

[0233] Copyright information 4302 is data indicating informationrelating to the copyright of a copyright holder of the object.

[0234] A copy permission information 4303 is data indicating whether ornot video information in a range contained in the figure approximatingthe object is permitted to be cut and re-used.

[0235] A foot mark 4304 is data indicating the time at which the objecthas finally been edited.

[0236] URL 4305 of related information formed by expressing data towhich a reference must be made when related information of the object isdisplayed by using URL.

[0237] Access limit information 4306 is data about informationpermission/inhibition of audience and jump owing to a hyper link foreach object and data for setting permission condition.

[0238] Billing information 4307 is data indicating billing informationfor each object.

[0239] Annotation data 4308 is data for aiding related information ofthe object and the operation.

[0240] Since a relatively small number of related information itemsshown in FIGS. 25 and 26 exists, it is preferable that relatedinformation is described in object region data.

[0241] A method of providing video data and object region data will nowbe described.

[0242] When object region data produced owing to the process accordingto this embodiment is provided for a user, a creator must provide objectregion data for the user by a method of some kind. The object regiondata may be provided by any one of the following methods.

[0243] (1) A method with which video data, its object region data andits related information are recorded in one (or a plurality of)recording medium so as to simultaneously be provided.

[0244] (2) A method with which video data and object region data arerecorded in one (or a plurality of) recording medium so as tosimultaneously be provided. However, related information is individuallyprovided or provision of the same is not performed (the latter case is acase in which related information can individually be acquired through anetwork or the like if provision is not performed).

[0245] (3) A method with which video data is solely provided. Moreover,object region data and related information are recorded in one (or aplurality of) recording medium so as to simultaneously be provided.

[0246] (4) A method with which video data, object region data andrelated information are individually provided.

[0247] The recording medium is mainly used to perform provision in theforegoing case. Another method may be employed with which a portion orthe overall portion of data is provided from a communication medium.

[0248] As described above, the structure according to this embodiment isable to describe the object region in a video as a parameter of a curveapproximating the time-sequential trajectory (the trajectory of thecoordinates of the representative points having the frame numbers ortime stamps as the variables) of the coordinates of the representativepoints of the approximate figure of the object region.

[0249] Since this embodiment enables the object region in a video to beexpressed with only the parameters of the function, object region data,the quantity of which can be reduced and which can easily be handled,can be generated. Moreover, expression of the representative points andgeneration of the parameters of the approximate curve can easily beperformed.

[0250] According to this embodiment, whether or not arbitrarycoordinates specified by a user indicate the object region canconsiderably easily be determined. Moreover, it leads to a fact thatspecification of a moving object in a video can furthermore easily beperformed.

[0251] Other embodiments of the object-region-data generating apparatusaccording to the present invention will be described. The same portionsas those of the first embodiment will be indicated in the same referencenumerals and their detailed description will be omitted.

[0252] Second Embodiment

[0253] The first embodiment has the structure that the representativepoints of a figure approximating the object region in a video isextracted so as to be converted into object region data. On the otherhand, a second embodiment has a structure that characteristic points inthe object region in the video are extracted so as to be converted intoobject region data.

[0254] Description will be made about the different structures fromthose according to the first embodiment.

[0255]FIG. 27 shows an example of the structure of an object-region-datagenerating apparatus according to this embodiment. As shown in FIG. 27,the object-region-data generating apparatus according to this embodimentincorporates a video data storage portion 230, a characteristic-pointextracting portion 233, a characteristic-point-curve approximatingportion 234 for approximating the arrangement of characteristic pointswith a curve, a related information storage portion 235 and a regiondata storage portion 236.

[0256] Referring to FIG. 27, a video data storage portion 230 has thesame function as that of the video data storage portion 100 according tothe first embodiment. The related information storage portion 235 hasthe same function as that of the related information storage portion 105according to the first embodiment. The region data storage portion 236has the same function as that of the region data storage portion 106according to the first embodiment.

[0257] The characteristic-point extracting portion 233 extracts at leastone characteristic point from the object region in the video. Thecharacteristic point may be any one a variety of points. For example,corners of an object (for example, a method disclosed in “Gray-levelcorner detection, L. Kitchen and A. Rosenfeld, Pattern RecognitionLetters, No. 1, pp. 95-102, 1982) or the centroid of the object may beemployed. When the centroid of the object is employed as thecharacteristic point, it is preferable that the portion around the pointassumed as the centroid is specified and then automatic extraction isperformed.

[0258] The characteristic-point-curve approximating portion 234 has abasic function similar to that of the representative point trajectorycurve approximating portion 104 according to the first embodiment. Thatis, the characteristic-point-curve approximating portion 234time-sequentially approximates, to a curve, the positions of thecharacteristic points extracted by the characteristic-point extractingportion 233. The approximate curve is, for each of the X and Ycoordinates, expressed as the function of the frame number f or the timestamp t given to the video so as to be approximated with a curve bylinear approximation or approximation using a spline curve. Data afterthe approximation has been performed is recorded by a method similar tothat according to the first embodiment.

[0259] Note that object region data according to this embodiment isbasically similar to object region data according to the firstembodiment (see FIG. 5). The field for the approximate figure number isnot required. Note that “data of the approximate figure” is “data ofcharacteristic points”.

[0260] Also data of the characteristic point in object region data isbasically similar to data of the approximate figure according to thefirst embodiment (see FIG. 6). Note that the “number of representativepoints” is the “number of characteristic points”. The “data of thetrajectory of representative points” is the “data of the trajectory ofcharacteristic points”. Note that figure type ID is not required.

[0261] Data of the trajectory of the characteristic points included inthe data of the characteristic points is similar to data of thetrajectory of the representative points according to the firstembodiment (see FIG. 7).

[0262]FIG. 28 is a flowchart showing an example of a flow of the processof the object-region-data generating apparatus according to thisembodiment. The overall flow is similar to that according to the firstembodiment. In step S21, video data for one frame is extracted from thevideo data storage portion 230 similarly to step S11 shown in FIG. 10.Steps S12 to S14 shown in FIG. 10 are made to be step S22 for extractingthe characteristic points of the object of interest. The approximationof the position of the representative point train of the approximatefigure in the successive frames with a curve in step S15 shown in FIG.10 is made to be step S23 for making approximation of the position ofthe characteristic point train of the object region in the successiveframes with a curve. Moreover, steps S24 and S25 are similar to stepsS16 and S17 shown in FIG. 10.

[0263] As a matter of course, the process according to this embodimentcan be realized by software.

[0264] As described above, the structure according to this embodiment isable to describe the object region in a video as a parameter of a curveapproximating the time-sequential trajectory (the trajectory of thecoordinates of the characteristic points having the frame numbers ortime stamps as the variables) of the characteristic points of theregion.

[0265] Since this embodiment enables the object region in a video to beexpressed with only the parameters of the function, object region data,the quantity of which can be reduced and which can easily be handled,can be generated. Moreover, expression of the characteristic points andgeneration of the parameters of the approximate curve can easily beperformed.

[0266] According to this embodiment, whether or not arbitrarycoordinates specified by a user indicate the object region canconsiderably easily be determined. Moreover, it leads to a fact thatspecification of a moving object in a video can furthermore easily beperformed.

[0267] Note that object region data based on the representative pointsof the approximate figure of the object region according to the firstembodiment and object region data based on the characteristic points ofthe object region according to the second embodiment may be mixed witheach other.

[0268] In the foregoing case, the format of object region data accordingto the first embodiment is provided with a flag for identifying a factthat object region data is based on the representative points of theapproximate figure of the object region or the characteristic points ofthe object region. As an alternative to providing the flag for theformat of object region data according to the first embodiment, when thefigure type ID has a specific value, a fact that object region data isbased on the characteristic points of the object region is indicated. Inthe other cases, a fact is indicated that object region data is based onthe representative points of the approximate figure of the objectregion.

[0269] The structure of object region data and a creating side have beendescribed. The portion for using the above-mentioned object region datawill now be described.

[0270] Third Embodiment

[0271] In the third embodiment, when object region data includingrelated information has been given to the object in the video, a userspecifies an object (mainly on a GUI screen) to display relatedinformation (display of characters, a still image or a moving image, oroutput of sound) or causes a related program to be executed.

[0272]FIG. 29 shows an example of the structure of a video processingapparatus according to this embodiment. As shown in FIG. 29, the videoprocessing apparatus according to this embodiment incorporates a videodata display portion 301, a control unit 302, a related informationdisplay portion 303 and at instruction input portion 304.

[0273] The video data display portion 301 displays video data input froma recording medium or the like (not shown) on a liquid crystal displayunit or a CRT.

[0274] The instruction input portion 304 permits a user to use apointing device, such as a mouse, or a keyboard to perform an operation,for example, specification of an object in the video displayed on theliquid crystal unit or the CRT. Moreover, the instruction input portion304 receives input (specification of an object) from the user.

[0275] The control unit 302, to be described later, determines whetheror not the user has specified the object in the video in accordancewith, for example, the coordinates specified by the user on the screenand object region data input from a recording medium (not shown).Moreover, the control unit 302 makes a reference to the pointer forpointing related information of object region data when a determinationhas been made that the user has specified a certain object in the video.Thus, the control unit 302 acquires related information of the object todisplay the related information.

[0276] The related information display portion 303 responds to theinstruction issued from the control unit 302 to acquire and displayrelated information (from a recording medium or a server or the likethrough a network).

[0277] When the pointer for pointing related information is a pointerfor pointing data in which program or the operation of the computer isdescribed, the computer performs a predetermined operation.

[0278] As a matter of course, also this embodiment may be realized bysoftware.

[0279] A process which is performed when the object region is expressedas an approximate figure similarly to the first embodiment will now bedescribed.

[0280]FIG. 30 shows an example of the process according to this example.The flowchart shown in FIG. 30 includes only a process which isperformed when a certain region in a video which is being displayedduring reproduction of the video is specified by using a pointingdevice, such as a mouse cursor (basically corresponding to the processwhich is performed by the control unit 302).

[0281] In step S31, the coordinates on the screen specified by using thepointing device or the like are calculated. Moreover, the frame numberof the video which is being reproduced at the moment of the instructionis acquired. Note that a time stamp may be employed as a substitute forthe frame number (hereinafter the frame number is employed).

[0282] In step S32, the object existing in the video having the framenumber in which the object has been specified is selected from objectregion data of the object added to the video. The foregoing selectioncan easily be performed by making a reference to the leading framenumber and the trailing frame number of object region data.

[0283] In step S33, data of a spline function (see FIGS. 6 and 7)extracted from object region data of the region selected in step S32 isused to calculate the coordinates of the representative points of theapproximate figure in the video display frame number when the object hasbeen specified. Thus, two focal points F and G and point H on theellipse are obtained in the example according to the first embodiment(see FIGS. 2 and 4).

[0284] In step S34, it is determined whether or not the coordinatesspecified by using the pointing device or the like exist in the object(that is, the approximate figure) in accordance with the discriminationprocedure which is decided according to the obtained representativepoints and the figure type ID of object region data.

[0285] When the ellipse is represented by the two focal points and onepoint on the ellipse similarly to the first embodiment, thedetermination can easily be made.

[0286] When, for example, the Euclidean distance between points P andpoint Q is expressed by E (P, Q) similarly to the first embodiment, thefollowing inequality is held in a case where the coordinate P specifiedby using the pointing device exists in the ellipse:

E(F, P)+E(G, P)≦E(F, H)+E(G, H)

[0287] In a case where the coordinate P exists on the outside of theellipse, the following inequality is held:

E(F, P)+E(G, P)>E(F, H)+E(G, H)

[0288] The foregoing inequalities are used to determine whether or notthe specified point exists in the object. Then, it is determined whetherstep S35 is performed or omitted (skipped) in accordance with a resultof the determination.

[0289] When a parallelogram is employed as the approximate figure of theobject region in the video, four inequalities are used as a substitutionfor one inequality in the case of the ellipse to determine whether ornot the arbitrary coordinates exist in the object.

[0290] When, for example, points A, B and C shown in FIG. 8 arerepresentative points, point D is obtained as follows:

D=C+A−B

[0291] Then, an assumption is made that a point on a straight linepassing through the points A and B is Q and the straight line isexpressed by the following equation:

f _(A,B)(Q)=0

[0292] When the point P exists in the figure, the following twoinequalities are simultaneously held:

f _(A,B)(P)×f _(C,D)(P)<0, and

f _(B,C)(P)×f _(D,A)(P)<0

[0293] where

[0294] f_(A,B)(P)=(y_(A)−y_(B))×(x−x_(A))−(x_(A)−x_(B))×(y−y_(A))

[0295] f_(B,C)(P)=(y_(B)−y_(C))×(x−x_(B))−(x_(B)−x_(C))×(y−y_(B))

[0296] f_(C,D)(P)=(y_(D)−y_(C))×(x−x_(C))−(x_(D)−x_(C))×(y−y_(C))

[0297] f_(D,A)(P)=(y_(A)−y_(D))×(x−x_(D))−(x_(A)−x_(D))×(y−y_(D)), and

[0298] P=(x, y), A=(x_(A), y_(A)), B=(x_(B), y_(B)), C=(x_(C), y_(C)),D=(x_(D), y_(D))

[0299] When approximation to one object with a plurality of approximatefigures is made (refer to the approximate figure number shown in FIGS. 5and 12), the foregoing process is performed for each approximate figure.

[0300] In step S35, a process which is performed only when the specifiedpoint exists in the object region. In the foregoing case, a reference tothe related information pointer 403 for pointing related informationcontained in object region data (see FIG. 5) is made. In accordance withinformation about the pointer, related information is acquired so as tobe, for example, displayed (in the example of the structure shown inFIG. 29, the foregoing process is performed by the related informationdisplay portion 303). When a program has been specified as relatedinformation, an specified program is executed or another specifiedoperation is performed. When related information has been described inobject region data, related information is required to be displayed.

[0301]FIG. 31 shows an example of a case where description (a text) ofan object in a video has been given as the related information. When thecoordinates specified by using the pointing device 802 duringreproduction of a video 800 exist in the object region 801 (a figureapproximating the object 801), related information 803 is displayed onan individual window.

[0302] In step S36, a branching process is performed so that it isdetermined whether or not an object having object region datafurthermore exists in the frame in which the object has been specified.If the object exists, the process proceeds to step S32. If the objectdoes not exist, the operation is completed.

[0303] When a plurality of regions overlap, either or both of theregions may arbitrarily be selected.

[0304] A process which is performed when the object region is expressedas characteristic points of the object similarly to the secondembodiment will now be described.

[0305] The portions different from those according to the firstembodiment will mainly be described.

[0306]FIG. 32 shows an example of the procedure according to thisexample. Note that the flowchart shown in FIG. 32 includes only aprocess (basically, corresponding to the process which is performed bythe control unit 302) which is performed when a certain region in avideo which is being displayed during reproduction of the video has beenspecified by using a pointing device, such as a mouse cursor. Since theoverall flow is similar to that of the flowchart shown in FIG. 30,different portions will mainly be described (steps S41, S42, S45 and S46are similar to steps S31, S32, S35 and S36).

[0307] In step S43, the coordinates of the position of thecharacteristic point of an object in a displayed frame number arecalculated from object region data. When a plurality of characteristicpoints exist, the coordinates of all of the characteristic points arecalculated.

[0308] In step S44, the distance between the position of thecharacteristic point calculated in step S43 and the coordinatesspecified by clicking is calculated for all of the characteristicpoints. Then, it is determined whether or not one or more characteristicpoint positioned distant for a distance shorter than a predeterminedthreshold value. Alternatively, a process for calculating the distancefor a certain characteristic point and comparing the distance with apredetermined threshold value is repeated. When one characteristic pointpositioned distant for a distance shorter than the threshold value isdetected, the process is interrupted. If one or more characteristicpoints distant for a distance shorter than the threshold value exits,the process proceeds to step S45. If no characteristic point of theforegoing type does not exist, the process proceeds to step S46.

[0309] As a result of the foregoing process, display of relatedinformation can be performed in accordance with the coordinates of thecharacteristic point of the object when a portion adjacent to the regionof the interest has been specified by an operation using a pointingdevice or the like.

[0310] Fourth Embodiment

[0311] A fourth embodiment will now be described with which an objectregion having related information which can be displayed is clearlydisplayed (communicated to a user) by using object region data. In theforegoing case, the object having related information which can bedisplayed must previously be supplied with object region data includinga pointer for pointing the related information.

[0312] The block structure of this embodiment is similar to thataccording to, for example, the third embodiment (see FIG. 29).

[0313] As a matter of course, also this embodiment can be realized bysoftware.

[0314] A case in which the object region has been expressed as anapproximate figure similar to the first embodiment will now bedescribed.

[0315]FIG. 33 shows an example of a process according to thisembodiment.

[0316] An example case in which the approximate figure is an ellipsewill now be described. As a matter of course, a similar process isperformed in a case of another approximate figure.

[0317] In step S51, the frame number of a video which is being displayedis acquired. Note that a time stamp may be employed as a substitute forthe frame number (hereinafter the frame number is employed).

[0318] In step S52, an object having the frame number acquired in stepS51 and existing in the video is selected. The selection is performed bydetecting data having a displayed frame number between the leading framenumber of object region data given to the video and the trailing framenumber.

[0319] In step S53, data of a spline function (see FIGS. 6 and 7) isextracted from object region data of the object selected in step S52.Then, the coordinates of representative points of an approximate figure(or a region having related information) in the displayed frame arecalculated.

[0320] In step S54, a reference to the figure type ID of object regiondata is made to obtain an approximate figure expressed by therepresentative points calculated in step S53. Then, display of the imagein each approximate figure (for example, an ellipse region) is changed.

[0321] The change may be performed by a variety of methods. When theapproximate figure is, for example, an ellipse, the brightness of theimage in the ellipse region is intensified by a predetermined value.Assuming that the degree of intensification is ΔY, the brightness beforethe change of the display is Y and an upper limit of the brightnesswhich can be displayed is Ymax, each pixel in the ellipse is displayedwith brightness of MIN(Y+ΔY, Ymax). Pixels on the outside of the ellipseare displayed with brightness of Y. Note that MIN(a, b) is a functiontaking a smaller value of a and b.

[0322]FIGS. 34A and 34B show an example with which the object region isdisplayed by the method with which the brightness is intensified (inFIGS. 34A and 34B, hatching indicates no change in the brightness and nohatching indicates intensified brightness). FIG. 34A shows a screen 1000which is in a state in which the display change process in step S54 hasnot been performed. Reference numeral 1001 represents an object havingobject region data in the video. A screen 1002 shown in FIG. 34B isdisplayed after the change in the display in step S54 has beenperformed. Reference numeral 1003 represents an ellipse regionapproximating the object region 1001. Display of only the inside portionof the ellipse region 1003 is brightened. Thus, a fact that the objectis an object which permits display or the like of related informationcan be recognized.

[0323] When one object is approximated with a plurality of approximatefigures (refer to approximate figure number shown in FIGS. 5 and 12),the foregoing process is performed for each approximate figure.

[0324] In step S55, it is determined whether or not another object, thedisplay of which must be changed, exists. A determination is madewhether or not a non-processed object having a display frame numberwhich is between the leading frame number and the trailing frame numberexists. If the non-processed object exists, the process from step S52 isrepeated. If no object of the foregoing type exists, the process iscompleted.

[0325] As described above, display of an object region having therelated information among the regions of the object in the video whichis specified by using object region data is changed. Thus, whether ornot the related information exists can quickly be detected.

[0326] A method of indicating the object region which permits display orthe like of related information may be the above-mentioned method withwhich the brightness in the object region is changed. Any one of avariety of methods may be employed. A variety of the methods will now bedescribed. The procedure of each process using object region data isbasically similar to the flowchart shown in FIG. 33. Therefore, step S54is changed to a corresponding process.

[0327] A display method shown in FIG. 35 is a method of displaying theposition of an object having related information on the outside of animage 1600. Reference numerals 1601 and 1602 represent objects havingrelated information. Reference numerals 1603 and 1604 represent bars fordisplaying the position of the object in the direction of the axis ofordinate and in the direction of the axis of abscissa. Display 1605 anddisplay 1606 correspond to the object 1601 having related information.FIG. 35 shows a structure that bars serving as marks are displayed inthe regions in which the region 1601 are projected in the direction ofthe axis of ordinate and in the direction of the axis of abscissa.Similarly, reference numerals 1607 and 1608 represent bars fordisplaying the object region 1602.

[0328] A state of projection of the object region in the foregoingdirections can easily be obtained by using the coordinates of therepresentative points of the approximate figure in the frame obtainedfrom data of the approximate figure of object region data and the figuretype ID as described in the embodiments.

[0329] It is preferable that the region of a different object isindicated with a bar displayed in a different manner (for example, adifferent color).

[0330] The method according to this embodiment causes a user to specifythe inside portion of the image in accordance with the bars 1603 and1604 displayed in the vertical and horizontal directions on the outsideof the image 1600 by using a pointing device. Thus, related informationcan be displayed.

[0331] It is preferable that the region of a different object isindicated with a bar displayed in a different manner (for example, adifferent color).

[0332]FIG. 36 shows another display method with which the position of anobject having related information is displayed on the outside of animage 1700. Objects 1701 and 1702 each having related information existin the image 1700. The position of the object having related informationis indicated by an object-position indicating bars 1703 and 1704. Asdistinct from the example shown in FIG. 35, each display bar indicatesonly the position of the centroid of the object as a substitute for theobject region. Circles 1705 and 1706 indicate the centroid of the object1701. Circles 1707 and 1708 indicate the centroid of the object 1702.

[0333] Also the centroid of the object region in the foregoingdirections can easily be obtained in accordance with the coordinates ofthe representative point of the approximate figure in the frame obtainedfrom data of the approximate figure of object region data and the figuretype ID.

[0334] The foregoing method enables display which can easily berecognized because the size of display on the object position indicatingbar can be reduced if the object has a large size or many objects exit.

[0335]FIG. 37 shows an example of a display method with which a relatedinformation list is displayed on the outside of an image 1800. The image1800 contains objects 1801 and 1802 each having related information.Reference numeral 1803 represents a list of objects each having relatedinformation. The list 1803 shows information of objects each havingrelated information in the image frame which is being displayed. In theexample shown in FIG. 37, names of objects are displayed which areobtained as a result of retrieving related information from objectregion data of the objects existing in the frame.

[0336] The foregoing method permits a user to cause related informationto be displayed by specifying the name shown in the related informationlist 1803 as well as the specifying the region 1801 or 1802 with thepointing device. Since also instruction of the number shown in the list1803 enables related information to be displayed, the foregoingstructure can be employed in a case of a remote control having nopointing device.

[0337]FIG. 38 shows a display method with which objects 1901 and 1902existing in an image 1900 and each having related information areindicated with icons 1903 and 1904 to indicate existence of relatedinformation. The icon 1903 corresponds to the object 1901, while theicon 1904 corresponds to the object 1902.

[0338] Each icon can be displayed by obtaining an approximate figure asdescribed above, by cutting a rectangle region having a predeterminedsize including the obtained approximate figure from video data in theframe and by, for example, arbitrarily contracting the cut rectangleregion.

[0339] The foregoing method enables related information to be displayedby directly specifying the icon as well as specifying the object regionin the video.

[0340]FIG. 39 shows an example of a display method configured to displaya map indicating the object region having related information so as toindicate existence of related information. An image 2000 includesobjects 2001 and 2002 each having related information. Reference numeral2003 represents a map of the regions of the objects each having relatedinformation. The map 2003 indicates the positions of the regions of theobjects each having related information in the image 2000. Referencenumeral 2004 represents the object 2001, while reference numeral 2005represents the object 2002.

[0341] The map 2003 has a form obtained by reducing the image 2000 andarranged to display only the images of the object regions (only theapproximate figures obtained as described above are displayed at thecorresponding positions in the contracted image).

[0342] The foregoing method enables related information to be displayedby specifying the object region 2004 or 2005 displayed on the map 2003as well as direct specification of an object in the image 2000.

[0343]FIGS. 40A and 40B show an example of the display method with whichspecification of an object existing in the image and having relatedinformation is facilitated by using a pointing device by controllingreproduction rate of the image at the position of the mouse cursor.Reference numerals 2100 and 2102 represent the overall bodies of thedisplay screens and reference numerals 2101 and 2103 represent regionson the display screens on which images are being displayed. In thedisplay screen 2100 shown in FIG. 40A, a mouse cursor 2104 is positionedon the outside of the image 2101 so that the image is reproduced at anormal display rate (frame/second)(or information. The map 2003indicates the positions of the regions of the objects each havingrelated information in the image 2000. Reference numeral 2004 representsthe object 2001, while reference numeral 2005 represents the object2002.

[0344] The map 2003 has a form obtained by reducing the image 2000 andarranged to display only the images of the object regions (only theapproximate figures obtained as described above are displayed at thecorresponding positions in the contracted image).

[0345] The foregoing method enables related information to be displayedby specifying the object region 2004 or 2005 displayed on the map 2003as well as direct specification of an object in the image 2000.

[0346]FIGS. 40A and 40B show an example of the display method with whichspecification of an object existing in the image and having relatedinformation is facilitated by using a pointing device by controllingreproduction rate of the image at the position of the mouse cursor.Reference numerals 2100 and 2102 represent the overall bodies of thedisplay screens and reference numerals 2101 and 2103 represent regionson the display screens on which images are being displayed. In thedisplay screen 2100 shown in FIG. 40A, a mouse cursor 2104 is positionedon the outside of the image 2101 so that the image is reproduced at anormal display rate (frame/second) (or reproducing speed). In thedisplay screen 2102 shown in FIG. 40B, the mouse cursor 2105 exists inthe image region 2103. Therefore, display rate of the image is loweredor displayed image is frozen.

[0347] Another structure may be employed as a substitute for theabove-mentioned structure in which image display rate is always loweredor the displayed image is frozen when the mouse cursor has entered theimage region. That is, whether or not an object having relatedinformation exists in the frame is determined (determination is made bycomparing the frame number and the leading frame number/trailing framenumber with each other). If the object having related information existsin the frame, the image display rate is lowered or the displayed imageis frozen.

[0348] For example, an object which is moving at high speed in the videocannot sometimes easily be specified by using the mouse cursor. Theforegoing method is arranged to change the reproducing speed accordingto the position of the mouse cursor. Thus, movement of the object can beslowed when the object is specified or the displayed image can befrozen. Hence it follows that instruction can easily be performed.

[0349]FIG. 41 shows an example of the display method with which anobject existing in the image and having related information can easilybe specified by using the pointing device. Reference numeral 2500represents an image which is being reproduced. Reference numeral 2501represents a button for acquiring an image. When the button 2501 isdepressed with a mouse pointer 2502, an image which has been displayedat the specified time can be acquired so as to be displayed on anacquired-image display portion 2503. The image 2500 is continuouslyreproduced even after the foregoing instruction has been performed withthe button 2501. Since the acquired image is displayed on theacquired-image display portion 2503 for a while, instruction of anobject which is being displayed in the acquired-image display portion2503 enables related information of the specified object to bedisplayed.

[0350] The button 2501 for acquiring an image may be omitted. Astructure may be employed from which the button 2501 is omitted and withwhich an image can automatically be acquired when the mouse cursor 2502enters the video display portion 2500.

[0351] A structure may be employed with which whether or not an objecthaving related information exists in the frame is determined when thebutton 2501 has been depressed or the mouse cursor has entered the imageregion (for example, a determination is made by comparing the framenumber and the leading frame number/trailing frame number with eachother). Only when the object having related information exists in theframe, the image is acquired so as to be displayed.

[0352] The foregoing method enables related information to easily bespecified from a still image which is being displayed on theacquired-image display portion 2503.

[0353] The foregoing variations may be employed. Another method may beemployed with which the region of an image which permits display or thelike of related information is clearly displayed. Also a method may beemployed with which instruction is facilitated. Thus, a variety ofmethods for aiding the operation of the user may be employed.

[0354] A case in which the object region is expressed as characteristicpoints of the object similarly to the second embodiment will now bedescribed.

[0355] Portions different from those according to the first embodimentwill mainly be described.

[0356] A flowchart is, in the foregoing case, a flowchart which isbasically similar to that shown in FIG. 33 except for characteristicpoints being employed as a substitute for the representative points.Specifically, the coordinates of characteristic points of theapproximate figure are calculated in step S53.

[0357]FIG. 34 shows the structure that the brightness in the approximatefigure 1003 corresponding to the object 1001 is intensified. If three ormore characteristic points exist in the foregoing case, a polygon havingthe vertices which are the characteristic points may be formed.Moreover, the brightness of the inside portion of the polygon may beintensified. If two or more characteristic points exist, a figure ofsome kind may be formed which has the representative points which arethe characteristic points. Moreover, the brightness in the figure may beintensified. Alternatively, a figure, such as a circle, the center ofwhich is each of the characteristic points and which has a somewhatlarge size is formed. Moreover, each of the formed figure, which must bedisplayed, is made conspicuous by means of brightness, color orblinking.

[0358] The structure shown in FIG. 35 is arranged such that projectionof the approximate figures corresponding to the objects 1601 and 1602 inthe vertical and horizontal directions is displayed as the bar set 1605and 1607 or the bar set 1606 and 1608. If three or more characteristicpoints exist in the foregoing case, a polygon having the vertices whichare the characteristic points may be formed. Moreover, projection of thepolygon in the directions of the two axes may be displayed as the bars.If two or more characteristic points exist, a rectangle having thevertices which are the characteristic points may be formed. Moreover,projection into the directions of the two axes may be displayed as thebars. If one characteristic point exists, the foregoing method shown inFIG. 36 may be employed with which the position of the centroid isdisplayed with circles in the bars.

[0359]FIG. 38 shows the structure with which the image of an object isextracted by cutting in accordance with the approximate figure or thelike so as to be displayed as an icon. Also in the foregoing case, theimage of an object can be extracted by cutting in accordance with thecharacteristic points so as to be displayed as an icon.

[0360]FIG. 39 shows a structure that the approximate figures 1903 and1904 are displayed in a map. Also in the foregoing case, a figure ofsome kind formed in accordance with characteristic points as describedabove may be displayed as a map.

[0361] The methods shown in FIGS. 37, 40 and 41 may employed in theforegoing case.

[0362] The foregoing variations may be employed. Another method may beemployed with which the region of an image which permits display or thelike of related information is clearly performed. Also a method may beemployed with which instruction is facilitated. Thus, a variety ofmethods for aiding the operation of the user may be employed.

[0363] Fifth Embodiment

[0364] A fifth embodiment will now be described with which an object ina video is retrieved.

[0365] The block structure according to this embodiment is similar tothat according to the third embodiment (see FIG. 29). Note that astructure shown in FIG. 29 may be arranged such that the relatedinformation display portion is omitted (for example, a system may beemployed with which retrieval of an object is performed without use ofrelated information). Another structure from which the instruction inputportion is omitted may be employed (for example, a structure may beemployed with which the GUI is not used to instruct the retrieval). As amatter of course, also this embodiment can be realized by software.

[0366] The third embodiment has the structure that the two focal pointsand one point on the ellipse are employed as the representative pointswhen the ellipse is employed. A structure will now be described in whichthree vertices of circumscribing rectangle of an ellipse are employed asthe representative points of an ellipse. As a matter of course, theretrieval is permitted regardless of employment of the representativepoints.

[0367] Note that the following symbols V₁ to V₄, P, Q, F₁, F₂, C₀, T, Uand C are vector quantities.

[0368] Since the present invention is configured to describe thetrajectory of the object region, estimation of points through which theobject has passed and points through which the object has not passedenables the object to be estimated. For example, retrieval such as“retrieve vehicles which have passed through the center of this crossingand entered that traffic lane” or “retrieve vehicles which have enteredthe road from this position and which have not moved to this trafficlane” can be performed.

[0369]FIGS. 42 and 43 show an example of the procedure for performingthe foregoing retrieval.

[0370]FIG. 42 shows an example of the procedure which is employed when arectangle is employed to express an object.

[0371] An assumption is made that point Q has been specified as thepoint through which the object has passed and has not passed.

[0372] In step S100, time at which an object has appeared at time t isset. In step S101, the coordinates of representative points V₁, V₂ andV₃ at certain time t are extracted. The coordinates are calculated asthe values of spline functions at the corresponding time. Thecoordinates of the residual vertices can easily be obtained inaccordance with the three vertices of the rectangle, as follows:

V ₄ =V ₁ −V ₂ +V ₃

[0373] In step S102, the values of four functions expressed by thefollowing equations are obtained.f₁(P) = (V_(2y) − V_(1y)) × (x − V_(1x)) − (V_(2x) − V_(1x)) × (y − V_(1y))f₂(P) = (V_(2y) − V_(3y)) × (x − V_(2x)) − (V_(2x) − V_(3x)) × (y − V_(2y))f₃(P) = (V_(3y) − V_(4y)) × (x − V_(3x)) − (V_(3x) − V_(4x)) × (y − V_(3y))f₄(P) = (V_(1y) − V_(4y)) × (x − V_(4x)) − (V_(1x) − V_(4x)) × (y − V_(4y))

[0374] where V_(i)=(V_(ix), V_(iy))

[0375] In step S103, it is determined whether or not the four obtainedp=(x, y) functions satisfy the following relationship:

f ₁(Q)×f ₃(Q)≦0 and f ₂(Q)×f ₄((Q)≦0

[0376] If the foregoing relationship is held, the object passes thespecified point Q at time t. Therefore, it is determined that the objectpasses through the point Q (step S104). If the relationship is not held,the object does not passes through the point Q at time t. Then, whetheror not the object has passed through the point Q at another time isdetected.

[0377] In step S105, it is determined whether or not detection of all ofmoments of time t has been performed by determining whether or not timet is the same as time at which the object has disappeared from thescreen. If the two moments of time are the same, the process iscompleted and it is determined that the object has not passed throughthe point Q (step S107). If time t is earlier than time at which theobject has disappeared, t is incremented by one in step S106. Then, theprocess from step S101 is repeated.

[0378] The foregoing process is performed for all of the objects whichis to be retrieved so that objects which satisfy the retrieval conditioncan be retrieved.

[0379]FIG. 43 shows an example of the procedure which is employed whenan ellipse is employed to express an object.

[0380] In step S110, time at which the object has appeared at time t isset.

[0381] In step S111 the coordinates of representative points V₁, V₂ andV₃ of the ellipse at certain time t are extracted. The representativepoints are the three vertices of the circumscribing rectangle of theellipse which are successively and clockwise arranged in an order as V₁,V₂ and V₃. The calculation is performed by a process similar to thatemployed to process the rectangle.

[0382] In step S112, a, b and points F₁ and F₂ expressed by thefollowing equations are obtained (F₁ and F₂ are obtained as followsaccording to the relationship in the magnitude between a and b):

a=|V ₂ −V ₁|/2

b=|V ₂ −V ₃|/2 $\begin{matrix}{F_{1} = \quad {C_{0} + {{e\left( {V_{2} - V_{1}} \right)}/2}}} & \quad & \quad & \left( {{w\quad h\quad e\quad n\quad a} > b} \right) \\{\quad {C_{0} + {{e\left( {V_{2} - V_{3}} \right)}/2}}} & \quad & \quad & \left( {{w\quad h\quad e\quad n\quad a} \leqq b} \right) \\{F_{2} = \quad {C_{0} - {{e\left( {V_{2} - V_{1}} \right)}/2}}} & \quad & \quad & \left( {{w\quad h\quad e\quad n\quad a} > b} \right) \\{\quad {C_{0} - {{e\left( {V_{2} - V_{3}} \right)}/2}}} & \quad & \quad & \left( {{w\quad h\quad e\quad n\quad a} \leqq b} \right)\end{matrix}$

[0383] where C₀ and e are as follows (e is determined in accordance withthe relationship in the magnitude between a and b)

C ₀=(V ₁ +V ₃)/2 $\begin{matrix}{e = \quad {\left\{ \sqrt{\left( {a^{2} - b^{2}} \right)} \right\}/a}} & \quad & \quad & \left( {{w\quad h\quad e\quad n\quad a} > b} \right) \\{\quad {\left\{ \sqrt{\left( {b^{2} - a^{2}} \right)} \right\}/b}} & \quad & \quad & \left( {{w\quad h\quad e\quad n\quad a} \leqq b} \right)\end{matrix}\quad$

[0384] In step S113, it is determined whether or not the followingconditions are satisfied (the conditions vary according to therelationship in the magnitude between a and b).

[0385] condition when a>b:

|F ₁ −Q|+|F ₂−Q|≦2a

[0386] condition when a≦b:

|F ₁ −Q|+|F ₂ −Q|≦2b

[0387] When the conditions are satisfied, the point Q exists in theellipse at time t. Therefore, it is determined that the object haspassed through the point Q and the process is completed (step S114). Ifthe conditions are not satisfied, the point Q exists on the outside ofthe ellipse at time t. Therefore, a similar process is performed forother moments of time t.

[0388] In step S115, it is determined as the completion conditionwhether or not time t is time at which the object has disappeared. Iftime t is time at which the object has disappeared, it is determinedthat the object has not passed through the point Q. Thus, the process iscompleted (step S117). If time t is not time at which the object hasdisappeared, t is incremented in step S116 and the process from stepS111 is repeated.

[0389] The foregoing process is performed for all of the objects whichis to be retrieved so that the objects which satisfy the retrievalconditions are retrieved.

[0390] The foregoing process is arranged such that a fact whether or notthe specified point is included in the approximate figure is employed asthe criterion for making determination. A variety of criteria may beemployed. For example, it may be determined that the object has passedthe point when the specified point exists adjacent to the approximatefigure. Alternatively, it may be determined when the specified pointsare successively included in the approximate figure over a predeterminednumber of frames.

[0391] Also in a case where another figure is employed to express theshape of the object, a process corresponding to the figure is performed.Thus, objects which satisfy the retrieval conditions can be retrieved.

[0392] When a plurality of points of passage or a plurality of point ofnon-passage have been specified, the foregoing process is performed forall of the specified points.

[0393] As a matter of course, one or more points of passage and one ormore points of non-passage may be combined with each other.

[0394] The retrieval can be performed by using the combinational logicfor a plurality of points of passage and points of non-passage. Forexample, retrieval can be performed, for example, “retrieve objectswhich have passed through either of point a or b and which have notpassed through both of points c and d”.

[0395] The retrieval of the point of passage can be widened to astructure that time for which the object exists at the point of passage.The foregoing retrieval includes “retrieve persons which have done freebrowsing for 10 minutes or longer” and “retrieve persons who were infront of the cash dispenser for three minutes or longer”. To perform theforegoing retrieval, time for which the object exists at the inputposition is measured. Then, only the objects which exist at the inputposition for time longer than time specified by the user are shown.

[0396] Another example of the widened retrieval will now be described inwhich a condition in terms of the size (the area of the object) isadded.

[0397] When the shape of the object is expressed by a rectangle or anellipse, the area of the object at certain time t can be calculated asfollows:

[0398] in the case of the rectangle,

S _(R) =|V ₂ −V ₁ |×|V ₃ −V ₂|

[0399] in the case of the ellipse,

S _(E) =abπ

[0400] When the obtained value is used, retrieval can be performed byusing a condition that, for example, the area is not smaller than S_(S)nor larger than S_(L). For example, when “retrieve persons which walk onthe road. Note that dogs and cats are not retrieved” is required,previous instruction of an area larger than that of the dogs and catsenables the retrieving accuracy to be improved.

[0401] Another example of the retrieval will now be described with whichobjects which have moved through similar trajectories are retrieved.

[0402] An assumption is made that the trajectories of a first object anda second object are T and U, respectively. Another assumption is madethat time for which the first object exists and time for which thesecond object exists are N_(T) and N_(U), respectively. An assumption ismade that N_(T)≧N_(U) in the foregoing case. Another assumption is madethat time at which each of the objects has appeared is t=0. Theforegoing conditions can always be satisfied by changing T and U and byshifting the origin of the time axis.

[0403] In the foregoing case, distance d(T, U) between T and U isdefined as follows:${D\left( {T,U} \right)} = {\min\limits_{i,}{\sum\limits_{j = 0}^{N_{U}}{E^{2}\left( {{T(j)},{U\left( {j + i} \right)}} \right)}}}$0 ≦ i ≦ N_(T) − N_(U)

[0404] The coordinates of T at time t is expressed as T (t) and E(P, Q)shows Euclidean distance.

[0405] By using the distance between the trajectories, the distancebetween the trajectory of the object specified by the user and thetrajectory of another object is calculated for all of the other objects.Thus, the object exhibiting the shortest distance is displayed or theobjects exhibiting the short distances are displayed by the numberspecified by the user. Thus, the objects which draw similar trajectoriescan be retrieved.

[0406] Moreover, an object which draws a trajectory similar to atrajectory drawn by a user by an input device such as a mouse can beretrieved. In the foregoing case, the trajectory drawn by the user doesnot contain time information. Therefore, the direction between thetrajectories must be calculated by a method distinct from d(T, U).Therefore, the distance d′(T, U) between the trajectory T and thetrajectory U drawn by the user is calculated as follows:${d^{\prime}\left( {T,U} \right)} = {\sum\limits_{i}^{N_{P\quad U} - 1}{\min\limits_{0 \leqq j \leqq N_{T}}\quad {E^{2}\left( {{T(j)},U_{i}} \right)}}}$

[0407] The trajectory drawn by the user is expressed by dot sequenceU_(i)(0≦i<N_(PU)). Note that N_(PU) is the number of the dot sequences.One or more objects exhibiting the short distance are displayed asobjects each drawing the similar trajectory. Thus, retrieval can beperformed.

[0408] When the trajectory of the center of the object has beendescribed, objects exhibiting short distance d(T, U) is retrieved suchthat the trajectories are T and U. When only information of a rectangleapproximating the shape of the object or the trajectory of an ellipsecan be obtained, the trajectory of the center is estimated. Then, thedistance between the trajectories of the objects is calculated. Anestimated value of center C at certain time t is obtained from thecoordinates V₁, V₂ and V₃ of the vertices of the rectangle or theellipse as follows:

C≈(V ₁ +V ₃)/2

[0409] As a result of the estimation, similar trajectories can beretrieved from the trajectories of all of the objects.

[0410] Although the example has been described in which therepresentative points of the approximate figure of the object region areemployed, the present invention may be applied to a case where thecharacteristic points of the object region are employed similarly to thesecond embodiment. In the foregoing case, whether or not the object haspassed through the specified point is determined in accordance with afact whether or not the distance between the characteristic point andthe specified point is shorter than a reference value.

[0411] The foregoing embodiments and structures may arbitrarily becombined with one another.

[0412] Each of the foregoing structures may be realized by a recordingmedium storing a program for causing a computer to execute apredetermined means (or causing the computer to act as a predeterminedmeans or causing the computer to realize a predetermined function).

[0413] Additional advantages and modifications will readily occur tothose skilled in the art. Therefore, the present invention in itsbroader aspects is not limited to the specific details, representativedevices, and illustrated examples shown and described herein.Accordingly, various modifications may be made without departing fromthe spirit or scope of the general inventive concept as defined by theappended claims and their equivalents.

[0414] The present invention is configured such that the object regionin a video is described as the parameter of a function approximating thetrajectory obtained by arranging positional data of representativepoints of the approximate figure of the object region or thecharacteristic points of the object region in a direction in whichframes proceed. Therefore, the region of a predetermined object can bedescribed with a small quantity of data. Moreover, creation and handlingof data can easily be performed.

[0415] According to the present invention, a user is able to easilyinstruct an object a video and determine the object.

[0416] According to the present invention, retrieval of an object in avideo can easily be performed.

What is claimed is:
 1. A region data describing method for describing,over a plurality of frames, region data about a region of an arbitraryobject existing in a video, the region data describing methodcomprising: extracting position data of a representative point of anapproximate figure approximating the region or a characteristic point ofthe region from the plurality of frames; determining a functionapproximating a trajectory which links corresponding representativepoints or corresponding characteristic points of successive frames, thefunction being represented by a parameter; and describing the parameterof the function as the region data.
 2. The region data describing methodaccording to claim 1, further comprising describing informationspecifying a leading frame or a trailing frame of said plurality offrames as the region data.
 3. The region data describing methodaccording to claim 2, further comprising describing information of thetype of the approximate figure as the region data.
 4. The region datadescribing method according to claim 2, further comprising describinginformation of the number of the approximate figure as the region data.5. The region data describing method according to claim 1, wherein theparameter includes position data of knots of the trajectory andinformation specifying the trajectory used together with position dataof the knots of the trajectory.
 6. The region data describing methodaccording to claim 1, wherein a plurality of the representative pointsor the characteristic points are included in a certain frame, and theregion data includes information specifying correspondence among aplurality of said representative points or characteristic points in thecertain frame and a plurality of said representative points orcharacteristic points in an adjacent frame.
 7. The region datadescribing method according to claim 1, further comprising describingrelated information related to the object or information indicating amethod of accessing to the related information.
 8. A region datagenerating apparatus for generating region data about a region of anarbitrary object existing in a plurality of frames of a video, theregion data generating apparatus comprising: an extracting circuitconfigured to extract position data of a representative point of anapproximate figure approximating the region or a characteristic point ofthe region from the plurality of frames; a function determining circuitconfigured to determine a function approximating a trajectory whichlinks corresponding representative points or correspondingcharacteristic points of successive frames, the function beingrepresented by a parameter; and a describing circuit configured todescribe the parameter of the function as the region data.
 9. The regiondata generating apparatus according to claim 8, wherein said describingcircuit describes information specifying a leading frame or a trailingframe of said plurality of frames.
 10. The region data generatingapparatus according to claim 9, wherein said describing circuitdescribes information of the type of the approximate figure.
 11. Theregion data generating apparatus according to claim 9, wherein saiddescribing circuit describes information of the number of theapproximate figure.
 12. The region data generating apparatus accordingto claim 8, wherein the parameter includes position data of knots of thetrajectory and information specifying the trajectory and used togetherwith position data of the knots of the trajectory.
 13. The region datagenerating apparatus according to claim 8, wherein a plurality of therepresentative points or the characteristic points are included in acertain frame, and the region data includes information specifyingcorrespondence among a plurality of said representative points orcharacteristic points in the certain frame and a plurality of saidrepresentative points or characteristic points in an adjacent frame. 14.The region data generating apparatus according to claim 8, wherein saiddescribing circuit describes related information related to the objector information indicating a method of accessing to the relatedinformation.
 15. A storing medium storing a computer program fordescribing, over a plurality of frames, region data about a region of anarbitrary object existing in a video, the computer program comprising: afirst program code of extracting position data of a representative pointof an approximate figure approximating the region or a characteristicpoint of the region from the plurality of frames; a second program codeof determining a function approximating a trajectory which linkscorresponding representative points or corresponding characteristicpoints of successive frames, the function being represented by aparameter; and a third program code of describing the parameter of thefunction.
 16. The storing medium according to claim 15, wherein saidthird program code describes information specifying a leading frame or atrailing frame of said plurality of frames.
 17. The storing mediumaccording to claim 16, wherein said third program code describesinformation of the type of the approximate figure.
 18. The storingmedium according to claim 16, wherein said third program code describesinformation of the number of the approximate figure.
 19. The storingmedium according to claim 15, wherein the parameter is position data ofknots of the trajectory and information specifying the trajectory andused together with position data of the knots of the trajectory.
 20. Thestoring medium according to claim 15, wherein a plurality of therepresentative points or the characteristic points are included in acertain frame, and said third program code describes informationspecifying correspondence among a plurality of said representativepoints or characteristic points in the certain frame and a plurality ofsaid representative points or characteristic points in an adjacentframe.
 21. The storing medium according to claim 15, wherein said thirdprogram code describes related information related to the object orinformation indicating a method of accessing to the related information.22. The storing medium according to claim 15, wherein the region datacomprises identification information of the object, informationspecifying a leading frame and a trailing frame of said plurality offrames, information related to the object, information indicating amethod of accessing to the related information, information of thenumber of the approximate figure, and approximate figure informationwhich includes information of the type of the approximate figure, numberinformation of the representative point, and function data of a splinefunction approximating the trajectories of the representative pointwhich includes knot information, order information of the splinefunction, and coefficient information of the spline function.
 23. Thestoring medium according to claim 15, wherein the region data comprisesidentification information of the object, information specifying aleading frame and a trailing frame of said plurality of frames, relatedinformation related to the object, information indicating a method ofaccessing to the related information, and characteristic pointinformation which includes information of the number of thecharacteristic point and function data of a spline functionapproximating the trajectories of the characteristic point whichincludes knot information, order information of the spline function, andcoefficient information of the spline function.